A Pipboy-like terminal application in Guile: Part 2 - more on strings

Hello again. I'm back to talk about recreating some Pip Boy like terminal widgets in ncurses.

Last time, I started talking about the remedial problems of how to *render* a string for a character cell console, e.g., how to convert a logical string to a list of visual strings that can be displayed on a terminal.  So far, I said that you need to

• Put the string in Unicode NFC normalization, because NFC normalized strings are more likely to be prepared glyphs on a terminal and not overstruck glyphs.  Unicode normalization is discussed in http://www.unicode.org/reports/tr15/
• Convert tabs to spaces
• Replace almost all of the control characters or Private Use Area characters with replacement glyphs.  This hygiene is necessary since some unhandled control character could garble the terminal display.  Some control characters that don't get removed are the 5 line separators (CR, LF, NEL, PS, and LS) and the 8 bidirectional  formatting characters (LRE, RLE, LRO, RLO, PDF, LRI, RLI, FSI, PDI, LRM, RLM, and ALM) because we'll use them in a second.

So that just leaves

• Prepare to handle any right-to-left text in the string.  This is described in the Unicode Bidirectional Algorithm in http://www.unicode.org/reports/tr9/
• Do any Arabic shaping of the glyphs
• Wrap the string to a given number of character cells.  The Unicode line-breaking algorithm is described here: http://www.unicode.org/reports/tr14/tr14-35.html
• Pad the list of wrapped strings to left, center, or right aligned

Bidirectional text

First off, I'm not qualified to talk about this, but here's the highlights.

Most European languages are written left to right.  The big right-to-left alphabets are Arabic and Hebrew.  Here in the Los Angeles, I'd guess the most common right-to-left languages are probably Persian, Hebrew, Arabic, and Yiddish.

But Scheme strings are normally encoded in logical order: e.g. the beginning of a string is the part of the string that would be read first by a human.  If a string contained a line of French, the first character would be the left-most character to be displayed on a screen.  If it contained Arabic, the first character would be the right-most character to be displayed on a screen.

Terminal emulators generally take one of two strategies when given strings to display. They either

• Display the text from left-to-right regardless of the contents of the text
• Or, try to be context sensitive when they display the text, switching from left-to-right and right-to-left depending on the apparent language of the text

Weirdly, in a ncurses application, neither strategy is particularly helpful.  If a terminal does the former, it becomes the programmers' responsibility to convert the string from logical order to visual order.  If it does the latter, and you ask ncurses to write a string at a given (y,x) position, it is hard to know if that x is columns counted from the left or columns counted from the right.

Consider the following program, and its output on the Cygwin terminal.  It starts ncurses, prints a line of Latin text starting at column 30, row 1, and prints a word in the Hebrew alphabet starting at column 30, row 2.  Both lines are supposed to begin a column 30, but, the terminal tries to be helpful and makes the second line print in the 30th column from the right, which is unlikely to be the intention of the programmer.

(use-modules (ncurses curses))
(setlocale LC_ALL "")
(define win (initscr))
(addstr win "LATIN TEXT" #:x 30 #:y 1)
(addstr win (string #\ט\# ק\# ס\# ט ) #:x 30 #:y 2)
(refresh win)
(sleep 10)
(endwin)

So, what to do?

To complicate matters, Unicode has some explicit control characters that can be embedded in strings when one wants to explicitly state the directionality of all or part of the text.  They can be used to explicitly indicate the direction of a run of text, or override the current general direction of the text. The 8 bidirectional  formatting characters (LRE, RLE, LRO, RLO, PDF, LRI, RLI, FSI, PDI, LRM, RLM, and ALM) need to be interpreted.  See Unicode TR#9 for details.

One strategy is to

1. Use a library like GNU FriBiDi to convert the string from logical to visual order
2. Set the terminal program to *not* try to help with bidirectionalization.

So, the FriBiDi function fribidi_log2vis is the important function for this.  There needs to be a Guile function that wraps up the FriBiDi functionality.  There isn't one yet.

Arabic Shaping

Arabic shaping is another one of those topics about which I know almost nothing, but, here's the highlights.

The same Arabic letter can have a different glyph depending on where it appears in a word.  If it appears in the middle of a word, the glyph should join smoothly to its neighboring letters, like English cursive letters.  If it appears at the beginning or end of a word it has a different form.  And it might also look different when it is a single letter not in a word.

Some new terminal emulators are smart enough to do this shaping for you. If it detects an Arabic letter in the middle of a work, it uses the correct glyph.  But if you've asked your terminal to *not* help with bidirectionalization so that the behavior of the ncurses screen locations is still predictable, it is still going to do Arabic letter shaping for you?  I don't know.  It is a mess.

There are other complications.  There are Unicode controls that exist to encourage characters to be joined (ZWJ, for example) or to discourage it (ZWNJ).

In any case, if you need to do shaping manually instead of leaving it to your terminal emulator, again GNU FriBiDi is your goto library for this.  And someone needs to package that for Guile, too.

Line Length and Line Breaking

OK.  You have your string.  It is NFC, untabified, has no nasty control characters in it, and you've decided upon some strategy for bidirectional text.  Next up, we need to figure out how much screen real estate each string takes up, and whether the lines need to be wrapped.

For console programs, each character takes up zero, one, or two cells. Latin letters usually take one cell. Chinese, Japanese, and Korean letters usually take two, as in the following pic.

So how do you tell how many cells a glyph is going to take?  Basically the C library function wcwidth is the basis for this.  It will tell you if a codepoint has a glyph that takes up one cell or two.  Unicode has their own explanation over here: http://unicode.org/reports/tr11/

Guile needs a function to compute the screen width of a line of text.  I like u32_strwidth from GNU Libunistring, and that's a function that needs to be made available to Guile, too.

But this also has some problems.  Some characters have a width that is ambiguous, and should be 2 cells on a screen that mostly consists of CJK text, or should be 1 on a screen that mostly consists of Latin text.  Some example ambiguous width characters are some punctuation or math symbols.  Not all terminals agree on what to do about ambiguous width characters. But I'm not going to solve this problem.

But once one knows how many screen cells a line of text is going to take, it would seem fairly easy to then construct a line-breaking algorithm.  Line breaks need to be put in at all the explicit line breaks of the five line breaking characters CR, LF, NEL, PS, and LS. Remember that CR+LF counts as just a single line break.  And then lines that exceed a desired number of columns need to be broken at plausible locations at the end of words.

There are some other complications.  There are some Unicode characters that are there to prevent line breaking or encourage line breaking: various hyphens, soft hyphens, double hyphens, word joiners.

Hopefully this convinces you that line breaking is not really something you should treat casually.    See Unicode TR #14 for its generic line breaking algorithm.  It is actually quite complex.  In any case, I'm not going to engineer a console line-wrapping algorithm.  I like the one in GNU Libunistring called u32_width_linebreaks() described here. And, again, this isn't available to Guile yet, so I need to package that, too.

But if there were such a function, it would take a string and break it into a list of strings, where each element of the list had one screen line of text.

Alignment and Padding

Alright, now we're at the end of this process.  The last step is padding the string to get the desired alignment.

The default alignment for Latin console text should be to have an aligned left margin and a ragged right margin.  For RTL text is should be to have an aligned right margin and a ragged left margin.  So you need a function that tries to determine the general directionality of a paragraph.

If you want a paragraph of text to be right-aligned or center-aligned, you need to pad the strings on the left with spaces.  To know how many spaces to pad a line, you need to know how many cells a line occupies on the screen.  Again it all goes back to wcwidth and  u32_strwidth as described above.

Next

So I'll come back when I have all the above functionality in some library somewhere, and we'll finally be ready to put some green text in a green box.

A Pipboy-like terminal application in Guile: Part 1 - rendering strings

So I've been playing a ton of Fallout 4.  It is not as much fun as Fallout: New Vegas, but, it is pretty good.

For the uninitiated, the Fallout games are videogame RPGs that deal with a lone wanderer making the world a better place by helping out in a post-nuclear-war world.  Part of the unusual world-building in the game is type of technology that exists in this future.

Most computers in this future are monochrome green-screen terminals, much like the old DEC VT420 terminals in our world.

Well, I do happen to be the maintainer of guile-ncurses, a TUI library for programming in Scheme.  So I think I'll spend a few weeks seeing if I can recreate some of the visual elements from Fallout 4 in a text-user interface, and in the process demonstrate something of the process of creating a TUI.

The problem with ncurses, as anyone who has tried to use it will quickly realize, it that it is quite a low-level toolkit.  In the analogy with GNOME, it is not GTK, it is Cairo or GDK.

(Note that there is a higher level TUI toolkit built on ncurses called CDK, which is much easier to use if you want to do any rapid prototyping on a TUI.  There is no Guile library for this, yet, though.)

If you want to follow along with the code, well, there isn't any yet.  But soon there will be some at my Github. https://github.com/spk121/pip-tui.git

Today we're going to try something that sounds simple but absolutely is not. In fact it is so complicated that I'm only going to begin the topic today.

We're going to work on displaying some text in a character cell terminal!  I know, crazy, right?

In ASCII, this is quite easy, of course.  But, I'm going include some details involved in using Unicode text, even if I ultimately end up just using ASCII, because, it is an interesting and valuable topic.  If we have a Unicode string that we want to display in a character cell terminal, we need to *render* it, to decide which glyphs go into which cells.  Rendering a string -- converting a string to a list of glyphs that go into cells -- has roughly the following steps.

• The string should be in the NFC normalization.
• The string needs to have horizontal tabs replaced with spaces.
• The C0 control characters that will not be handled later should be replaced with replacement characters.  Only carriage return and line feed will be handled later.
• C1 controls, except for NEL, unassigned characters, and private-use characters should be replaced with the generic replacement character.
• Some strings have a different visual order than their logical order.  Hebrew and Arabic have a visual order that is right-to-left even though the strings are encoded in logical order, from beginning to end.  This is handled by the Unicode bidirectional algorithm.
• Arabic letters are cursive, so the various letters can have different shapes depending on if they appear at the beginning, the middle, or the end of a word.
• The rendered string has to be in the locale of the terminal to display properly.  Characters that cannot be properly displayed need to be replaced with replacement characters.
• And lastly, strings need to be line wrapped if they occupy more cells than are available.  This means understanding the number of cells a string occupies. On terminals, Unicode codepoints take zero, one, or two cells of space. Latin characters take one cell.  Fullwidth hanzi and kanji take two cells.

To begin, we start with the operations that need to occur in *logical order* not *visual order*.

Normalization

When rendering strings for use with character cell terminals, it is much easier to deal with composed glyphs, such as U+00C0 LATIN CAPITAL LETTER A WITH GRAVE, than with the combining accents, such as U+0041 LATIN CAPITAL LETTER A and U+0300 COMBINING GRAVE ACCENT.

All strings to be rendered should thus be passed through core Guile function string-normalize-nfc.

Untabifying

Canonically a horizontal tab occupies 8 spaces, so each instance of a horizontal tab in a string must be replace with 8 spaces.

Pedantry: on some old terminals and line printers, horizontal tabs actually moved one to the next tab stop, which was the next column that was a multiple of 8.

Here's one way to do the string expansion.

(define (string-untabify in-string TABSIZE)
  (string-fold
   (lambda (c str)
     (if (char=? c #\tab)
         (string-append str (make-string TABSIZE #\space))
         (string-append str (string c))))
   ""
   in-string))

Replacement Characters

Next up is replacing some of the control, private-use, and unrepresented characters with replacement symbols.  Why do this?

1. If you try to print C0 control characters (U+0000 to U+001F) to a console, the console might interpret it as commands.
2. If you try to print C1 control characters (U+0080 to U+009F) to a console, consoles in 8-bit mode might interpret them as commands, and can potentially hang.
3. Some characters -- such as all the unassigned characters and Private Use Area (PUA) characters -- won't be rendered by any character cell terminal, so there is no point in allowing them to continue.

To replace the C0 control codes from U+0000 to U+001F and also U+007F with graphical symbols, there are two reasonable choices. ISO-2047 symbols are fairly obscure but, hey, they are a standard.  But, the graphical pictures for control codes starting at U+2400 are more familiar and are probably a better choice.

Unicode control codes for U+0000 NULL to U+0007 BELL look like ␀␁␂␃␄␅␆␇.
ISO-2047 control codes for U+0000 NULL to U+0007 BELL look like ⎕⌈⊥⌋⌁⊠✓⍾.
There are many other codepoints that should just be replaced with �, the U+FFFD REPLACEMENT CHARACTER, including

• All the C1 control characters U+0080 to U+009F, excluding U+0085 NEXT LINE (NEL).
• All the private use area characters U+E000..U+F8FF, U+F0000..U+FFFFD, and U+100000..U+10FFFD
• Any other character with the general category of control, surrogate, or unassigned.

Note that we do not include the control format characters in this list, such as U+200E LEFT-TO-RIGHT MARK, because we'll need them later when be handle bi-directional text.

In the library, there will be a function that might be called string-replace-controls-and-pua which will do this replacement.

That's all for now.  Next time we'll handle bidirectional text, Arabic shaping, and line wrapping.