A Pipboy-like terminal application in Guile: Part 1 - rendering strings

So I've been playing a ton of Fallout 4.  It is not as much fun as Fallout: New Vegas, but, it is pretty good.

For the uninitiated, the Fallout games are videogame RPGs that deal with a lone wanderer making the world a better place by helping out in a post-nuclear-war world.  Part of the unusual world-building in the game is type of technology that exists in this future.

Most computers in this future are monochrome green-screen terminals, much like the old DEC VT420 terminals in our world.

Well, I do happen to be the maintainer of guile-ncurses, a TUI library for programming in Scheme.  So I think I'll spend a few weeks seeing if I can recreate some of the visual elements from Fallout 4 in a text-user interface, and in the process demonstrate something of the process of creating a TUI.

The problem with ncurses, as anyone who has tried to use it will quickly realize, it that it is quite a low-level toolkit.  In the analogy with GNOME, it is not GTK, it is Cairo or GDK.

(Note that there is a higher level TUI toolkit built on ncurses called CDK, which is much easier to use if you want to do any rapid prototyping on a TUI.  There is no Guile library for this, yet, though.)

If you want to follow along with the code, well, there isn't any yet.  But soon there will be some at my Github. https://github.com/spk121/pip-tui.git

Today we're going to try something that sounds simple but absolutely is not. In fact it is so complicated that I'm only going to begin the topic today.

We're going to work on displaying some text in a character cell terminal!  I know, crazy, right?

In ASCII, this is quite easy, of course.  But, I'm going include some details involved in using Unicode text, even if I ultimately end up just using ASCII, because, it is an interesting and valuable topic.  If we have a Unicode string that we want to display in a character cell terminal, we need to *render* it, to decide which glyphs go into which cells.  Rendering a string -- converting a string to a list of glyphs that go into cells -- has roughly the following steps.

• The string should be in the NFC normalization.
• The string needs to have horizontal tabs replaced with spaces.
• The C0 control characters that will not be handled later should be replaced with replacement characters.  Only carriage return and line feed will be handled later.
• C1 controls, except for NEL, unassigned characters, and private-use characters should be replaced with the generic replacement character.
• Some strings have a different visual order than their logical order.  Hebrew and Arabic have a visual order that is right-to-left even though the strings are encoded in logical order, from beginning to end.  This is handled by the Unicode bidirectional algorithm.
• Arabic letters are cursive, so the various letters can have different shapes depending on if they appear at the beginning, the middle, or the end of a word.
• The rendered string has to be in the locale of the terminal to display properly.  Characters that cannot be properly displayed need to be replaced with replacement characters.
• And lastly, strings need to be line wrapped if they occupy more cells than are available.  This means understanding the number of cells a string occupies. On terminals, Unicode codepoints take zero, one, or two cells of space. Latin characters take one cell.  Fullwidth hanzi and kanji take two cells.

To begin, we start with the operations that need to occur in *logical order* not *visual order*.

Normalization

When rendering strings for use with character cell terminals, it is much easier to deal with composed glyphs, such as U+00C0 LATIN CAPITAL LETTER A WITH GRAVE, than with the combining accents, such as U+0041 LATIN CAPITAL LETTER A and U+0300 COMBINING GRAVE ACCENT.

All strings to be rendered should thus be passed through core Guile function string-normalize-nfc.

Untabifying

Canonically a horizontal tab occupies 8 spaces, so each instance of a horizontal tab in a string must be replace with 8 spaces.

Pedantry: on some old terminals and line printers, horizontal tabs actually moved one to the next tab stop, which was the next column that was a multiple of 8.

Here's one way to do the string expansion.

(define (string-untabify in-string TABSIZE)
  (string-fold
   (lambda (c str)
     (if (char=? c #\tab)
         (string-append str (make-string TABSIZE #\space))
         (string-append str (string c))))
   ""
   in-string))

Replacement Characters

Next up is replacing some of the control, private-use, and unrepresented characters with replacement symbols.  Why do this?

1. If you try to print C0 control characters (U+0000 to U+001F) to a console, the console might interpret it as commands.
2. If you try to print C1 control characters (U+0080 to U+009F) to a console, consoles in 8-bit mode might interpret them as commands, and can potentially hang.
3. Some characters -- such as all the unassigned characters and Private Use Area (PUA) characters -- won't be rendered by any character cell terminal, so there is no point in allowing them to continue.

To replace the C0 control codes from U+0000 to U+001F and also U+007F with graphical symbols, there are two reasonable choices. ISO-2047 symbols are fairly obscure but, hey, they are a standard.  But, the graphical pictures for control codes starting at U+2400 are more familiar and are probably a better choice.

Unicode control codes for U+0000 NULL to U+0007 BELL look like ␀␁␂␃␄␅␆␇.
ISO-2047 control codes for U+0000 NULL to U+0007 BELL look like ⎕⌈⊥⌋⌁⊠✓⍾.
There are many other codepoints that should just be replaced with �, the U+FFFD REPLACEMENT CHARACTER, including

• All the C1 control characters U+0080 to U+009F, excluding U+0085 NEXT LINE (NEL).
• All the private use area characters U+E000..U+F8FF, U+F0000..U+FFFFD, and U+100000..U+10FFFD
• Any other character with the general category of control, surrogate, or unassigned.

Note that we do not include the control format characters in this list, such as U+200E LEFT-TO-RIGHT MARK, because we'll need them later when be handle bi-directional text.

In the library, there will be a function that might be called string-replace-controls-and-pua which will do this replacement.

That's all for now.  Next time we'll handle bidirectional text, Arabic shaping, and line wrapping.