10. Grapheme cluster breaks in strings <unigbrk.h>

This include file declares functions for determining where in a string “grapheme clusters” start and end. A “grapheme cluster” is an approximation to a user-perceived character, which sometimes corresponds to multiple Unicode characters. Editing operations such as mouse selection, cursor movement, and backspacing often operate on grapheme clusters as units, not on individual characters.

Some grapheme clusters are built from a base character and a combining character. The letter ‘é’, for example, is most commonly represented in Unicode as a single character U+00E8 LATIN SMALL LETTER E WITH ACUTE. It is, however, equally valid to use the pair of characters U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Since the user would perceive this pair of characters as a single character, they would be grouped into a single grapheme cluster.

But there are also grapheme clusters that consist of several base characters. For example, a Devanagari letter and a Devanagari vowel sign that follows it may form a grapheme cluster. Similarly, some pairs of Thai characters and Hangul syllables (formed by two or three Hangul characters) are grapheme clusters.

10.1 Grapheme cluster breaks in a string

The following functions find a single boundary between grapheme clusters in a string.

Function: void u8_grapheme_next (const uint8_t *s, const uint8_t *end)

Function: void u16_grapheme_next (const uint16_t *s, const uint16_t *end)

Function: void u32_grapheme_next (const uint32_t *s, const uint32_t *end)

Returns the start of the next grapheme cluster following s, or end if no grapheme cluster break is encountered before it. Returns NULL if and only if s == end.

Note that these functions do not handle the case when a character outside of the range between s and end is needed to determine the boundary. This is the case in particular with syllables in Indic scripts or emojis. Use _grapheme_breaks functions for such cases.

Function: void u8_grapheme_prev (const uint8_t *s, const uint8_t *start)

Function: void u16_grapheme_prev (const uint16_t *s, const uint16_t *start)

Function: void u32_grapheme_prev (const uint32_t *s, const uint32_t *start)

Returns the start of the grapheme cluster preceding s, or start if no grapheme cluster break is encountered before it. Returns NULL if and only if s == start.

Note that these functions do not handle the case when a character outside of the range between start and s is needed to determine the boundary. This is the case in particular with syllables in Indic scripts or emojis. Use _grapheme_breaks functions for such cases.

Note also that these functions work only on well-formed Unicode strings.

The following functions determine all of the grapheme cluster boundaries in a string.

Function: void u8_grapheme_breaks (const uint8_t *s, size_t n, char *p)

Function: void u16_grapheme_breaks (const uint16_t *s, size_t n, char *p)

Function: void u32_grapheme_breaks (const uint32_t *s, size_t n, char *p)

Function: void ulc_grapheme_breaks (const char *s, size_t n, char *p)

Function: void uc_grapheme_breaks (const ucs_t *s, size_t n, char *p)

Determines the grapheme cluster break points in s, an array of n units, and stores the result at p[0..nx-1].

p[i] = 1

means that there is a grapheme cluster boundary between s[i-1] and s[i].

p[i] = 0

means that s[i-1] and s[i] are part of the same grapheme cluster.

p[0] is always set to 1, because there is always a grapheme cluster break at start of text.

In addition to the above variants for UTF-8, UTF-16, and UTF-32 strings, <unigbrk.h> provides another variant: uc_grapheme_breaks.

This is similar to u32_grapheme_breaks, but it accepts any characters which may not be represented in UTF-32, such as control characters.

10.2 Grapheme cluster break property

This is a more low-level API. The grapheme cluster break property is a property defined in Unicode Standard Annex #29, section “Grapheme Cluster Boundaries”, see https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries. It is used for determining the grapheme cluster breaks in a string.

The following are the possible values of the grapheme cluster break property. More values may be added in the future.

Constant: int GBP_OTHER

Constant: int GBP_CR

Constant: int GBP_LF

Constant: int GBP_CONTROL

Constant: int GBP_EXTEND

Constant: int GBP_PREPEND

Constant: int GBP_SPACINGMARK

Constant: int GBP_L

Constant: int GBP_V

Constant: int GBP_T

Constant: int GBP_LV

Constant: int GBP_LVT

Constant: int GBP_RI

Constant: int GBP_ZWJ

Constant: int GBP_EB

Constant: int GBP_EM

Constant: int GBP_GAZ

Constant: int GBP_EBG

The following function looks up the grapheme cluster break property of a character.

Function: int uc_graphemeclusterbreak_property (ucs4_t uc)

Returns the Grapheme_Cluster_Break property of a Unicode character.

The following function determines whether there is a grapheme cluster break between two Unicode characters. It is the primitive upon which the higher-level functions in the previous section are directly based.

Function: bool uc_is_grapheme_break (ucs4_t a, ucs4_t b)

Returns true if there is an grapheme cluster boundary between Unicode characters a and b.

There is always a grapheme cluster break at the start or end of text. You can specify zero for a or b to indicate start of text or end of text, respectively.

This implements the extended (not legacy) grapheme cluster rules described in the Unicode standard, because the standard says that they are preferred.

Note that this function does not handle the case when three or more consecutive characters are needed to determine the boundary. This is the case in particular with syllables in Indic scripts or emojis. Use uc_grapheme_breaks for such cases.

This document was generated by Bruno Haible on February, 24 2024 using texi2html 1.78a.