A Pipboy-like terminal application in Guile: Part 1 - rendering strings

So I've been playing a ton of Fallout 4.  It is not as much fun as Fallout: New Vegas, but, it is pretty good.

For the uninitiated, the Fallout games are videogame RPGs that deal with a lone wanderer making the world a better place by helping out in a post-nuclear-war world.  Part of the unusual world-building in the game is type of technology that exists in this future.

Most computers in this future are monochrome green-screen terminals, much like the old DEC VT420 terminals in our world.

Well, I do happen to be the maintainer of guile-ncurses, a TUI library for programming in Scheme.  So I think I'll spend a few weeks seeing if I can recreate some of the visual elements from Fallout 4 in a text-user interface, and in the process demonstrate something of the process of creating a TUI.

The problem with ncurses, as anyone who has tried to use it will quickly realize, it that it is quite a low-level toolkit.  In the analogy with GNOME, it is not GTK, it is Cairo or GDK.

(Note that there is a higher level TUI toolkit built on ncurses called CDK, which is much easier to use if you want to do any rapid prototyping on a TUI.  There is no Guile library for this, yet, though.)

If you want to follow along with the code, well, there isn't any yet.  But soon there will be some at my Github. https://github.com/spk121/pip-tui.git

Today we're going to try something that sounds simple but absolutely is not. In fact it is so complicated that I'm only going to begin the topic today.

We're going to work on displaying some text in a character cell terminal!  I know, crazy, right?

In ASCII, this is quite easy, of course.  But, I'm going include some details involved in using Unicode text, even if I ultimately end up just using ASCII, because, it is an interesting and valuable topic.  If we have a Unicode string that we want to display in a character cell terminal, we need to *render* it, to decide which glyphs go into which cells.  Rendering a string -- converting a string to a list of glyphs that go into cells -- has roughly the following steps.

• The string should be in the NFC normalization.
• The string needs to have horizontal tabs replaced with spaces.
• The C0 control characters that will not be handled later should be replaced with replacement characters.  Only carriage return and line feed will be handled later.
• C1 controls, except for NEL, unassigned characters, and private-use characters should be replaced with the generic replacement character.
• Some strings have a different visual order than their logical order.  Hebrew and Arabic have a visual order that is right-to-left even though the strings are encoded in logical order, from beginning to end.  This is handled by the Unicode bidirectional algorithm.
• Arabic letters are cursive, so the various letters can have different shapes depending on if they appear at the beginning, the middle, or the end of a word.
• The rendered string has to be in the locale of the terminal to display properly.  Characters that cannot be properly displayed need to be replaced with replacement characters.
• And lastly, strings need to be line wrapped if they occupy more cells than are available.  This means understanding the number of cells a string occupies. On terminals, Unicode codepoints take zero, one, or two cells of space. Latin characters take one cell.  Fullwidth hanzi and kanji take two cells.

To begin, we start with the operations that need to occur in *logical order* not *visual order*.

Normalization

When rendering strings for use with character cell terminals, it is much easier to deal with composed glyphs, such as U+00C0 LATIN CAPITAL LETTER A WITH GRAVE, than with the combining accents, such as U+0041 LATIN CAPITAL LETTER A and U+0300 COMBINING GRAVE ACCENT.

All strings to be rendered should thus be passed through core Guile function string-normalize-nfc.

Untabifying

Canonically a horizontal tab occupies 8 spaces, so each instance of a horizontal tab in a string must be replace with 8 spaces.

Pedantry: on some old terminals and line printers, horizontal tabs actually moved one to the next tab stop, which was the next column that was a multiple of 8.

Here's one way to do the string expansion.

(define (string-untabify in-string TABSIZE)
  (string-fold
   (lambda (c str)
     (if (char=? c #\tab)
         (string-append str (make-string TABSIZE #\space))
         (string-append str (string c))))
   ""
   in-string))

Replacement Characters

Next up is replacing some of the control, private-use, and unrepresented characters with replacement symbols.  Why do this?

1. If you try to print C0 control characters (U+0000 to U+001F) to a console, the console might interpret it as commands.
2. If you try to print C1 control characters (U+0080 to U+009F) to a console, consoles in 8-bit mode might interpret them as commands, and can potentially hang.
3. Some characters -- such as all the unassigned characters and Private Use Area (PUA) characters -- won't be rendered by any character cell terminal, so there is no point in allowing them to continue.

To replace the C0 control codes from U+0000 to U+001F and also U+007F with graphical symbols, there are two reasonable choices. ISO-2047 symbols are fairly obscure but, hey, they are a standard.  But, the graphical pictures for control codes starting at U+2400 are more familiar and are probably a better choice.

Unicode control codes for U+0000 NULL to U+0007 BELL look like ␀␁␂␃␄␅␆␇.
ISO-2047 control codes for U+0000 NULL to U+0007 BELL look like ⎕⌈⊥⌋⌁⊠✓⍾.
There are many other codepoints that should just be replaced with �, the U+FFFD REPLACEMENT CHARACTER, including

• All the C1 control characters U+0080 to U+009F, excluding U+0085 NEXT LINE (NEL).
• All the private use area characters U+E000..U+F8FF, U+F0000..U+FFFFD, and U+100000..U+10FFFD
• Any other character with the general category of control, surrogate, or unassigned.

Note that we do not include the control format characters in this list, such as U+200E LEFT-TO-RIGHT MARK, because we'll need them later when be handle bi-directional text.

In the library, there will be a function that might be called string-replace-controls-and-pua which will do this replacement.

That's all for now.  Next time we'll handle bidirectional text, Arabic shaping, and line wrapping.

A Portrait of the Coder as an Old Man

I've been thinking quite a bit about being a older coder.  I've come to the realization that, like most other professions or activities, as we age, we need to make accommodations.

For example, as we age, the short term memory begins to fade.  Where once I could reliably recall the names and type details of hundreds of the variables, functions, and methods I've touched on a large task, now I can only reliably remember a couple dozen.  As a young man, I could focus on whatever function I was modifying, using memory and recall should I need to add a function call or variable reference.  Now I do spend a lot of time tap dancing through code, always referencing headers and declarations, with dozens of files open at any given moment.

On the flip side, the long term memory, now so richly loaded with associations, can get bogged down in those associations.  Every module or algorithm brings to mind another module or algorithm.  This is like that is like this is like that.  Which is very useful given that there is nothing new under the sun, but sometimes there is something new that needs to be considered on its own merits.

Also, there is the problem of irascibility.

I have spend almost all of my professional life coding in subsets of C.  Various embedded or safe or DO-178B restrictions to C.

C, in particular, has two problems for the old coder.

First is the problem of irascibility.

C is very simple with a minimal library.  I remember the first time I coded a doubly-linked list or a binary tree.  So satisfying to get this concept and see it function.  It is fun the first time, or the second.  But after 30 years, I never again want to code a doubly-linked list, a sort, a hash map, a binary tree, a dispatcher, a parser, a lexer, a matrix pseudoclass, a polynomial solver, or a Kalman filter.  It is just no fun anymore.

Second is the problem of verbosity.

Since I know I no longer have an infallible, photographic recall of an entire codebase, it helps to be able to have as much code on the screen at any given moment: to be able to see as much as possible of what I am doing.  And C, in particular, is so verbose that it restricts the amount of meaning I can throw up on any given screen.  C so often oscillates between being information dense and having a paucity of information per line.

So, then, coding as an old man...

• I need my language and tools to help me remember details of variables, functions, and methods.  And I need those tools to help me remember in a way that doesn't destroy my flow.
• I need my language and tools to let me see as much of what I am doing as possible.
• I need my language and tools to not require me to re-code or re-integrate fundamental algorithms that are no longer entertaining.
• I need my language and tools to help me determine if a code block is or is not a fundamental algorithm.  If it is, I want to use that fundamental algorithm.

Let's imagine, for a second, the properties of a theoretical programming language created for older coders. Such a theoretical language would be designed from the ground to play well with tools.  There should be nothing in the grammar of the language that would prevent tools from autocompleting or suggesting a given declaration.  The scoping and context rules should not have the sort of ambiguity that prevents effective tooling.  And compilation errors should be flagged immediately.

Also such a language should have a consistent level of information density, so that a screenful of text is sufficient to display a "unit" of code.

I don't often stray too far from the C-language family.  To be honest, C# is pretty good as an old man language. But, I've been trying to brush up on my C++17.

#include 〈algorithm〉 is pretty cool.