The encoding of MAM for Sefaria

Author: Ben Denckla

Revision: 2021-09-08 / ב׳ בְּתִשְׁרֵי תשפ״ב

Introduction

This document describes the encoding of MAM (Miqra According to the Masorah) for Sefaria. MAM is now the default Tanakh on Sefaria. Rather than describing the encoding in full, this document focuses only on differences in encoding between MAM and Sefaria’s former default Tanakh. Before we go any further, some terminology:

1. By “MAM” I usually mean, specifically, the CSV export of the Google Sheet for MAM. This CSV export is intended for import into the Sefaria system. To be clear, this CSV export is not generated by Google Sheets. It is generated by a Python program I wrote. That software that operates on the data contained in the Google Sheet for MAM.
2. By “old Sefaria” I usually mean, specifically, Sefaria’s former default Tanakh. This Tanakh was originally imported from what is now called UXLC: the tanach.us Unicode XML version of the Groves WLC.

This document has three top-level sections:

1. Tag encoding: This section describes features encoded using HTML tags like <span> & <small>. Some of these features are not solely encoded using tags: string details are sometimes relevant, too. Thus, there is some conceptual overlap between this section & the next, which describes features solely encoded at the string level. This section includes the following subsections:
   1. List of class attribute values
   2. Legarmeih & פסק
   3. Small, large, & hung letters
   4. Samekh, pe, & inverted nun
   5. Implicit maqaf
   6. The five main ketiv/qere types
   7. Ketiv velo qere & qere velo ketiv
   8. Trivial ketiv/qere pairs
   9. Notes
   10. Addendums
2. String encoding: This section describes features encoded solely at the most basic level: Unicode strings. This section includes the following subsections:
   1. Qamats qatan
   2. אתנח הפוך
   3. גרש-telisha & gershayim-telisha
   4. Stress זרקא
   5. No displacement of זרקא by ל
   6. No special meteg placements
   7. Robust Unicode
3. CSV topics: This section includes the following subsections:
   1. Metadata fields: This subsection describes MAM’s values for selected metadata fields.
   2. Joshua 21: This subsection describes how MAM encodes two verses that it lacks compared to old Sefaria.

This document uses the following typographic conventions:

1. Text styled like this will hover-reveal some extra explanation (only in media supporting hovering, of course). These hover-reveals may even be possible when viewing the PDF version of this document, depending on which viewer is being used.
2. Text styled like this is intended to focus the reader’s attention to certain letters within a word. E.g. ב in אבג.
3. Text styled like this is the name of a code point in Unicode. These names are often abbreviated, e.g. ZARQA is used instead of HEBREW ACCENT ZARQA. Some abbreviations like CGJ can hover-expanded (only in media supporting hovering, of course).

Some further notes on the names of code points:

1. Some Hebrew marks do not have a correspondingly-named code point. Though this situation may seem undesirable, on the bright side, we can safely use the mark’s name without risk of confusion about whether we are referring to a code point. In such cases it is clear that the mark’s name refers to the more general, diffuse, and abstract idea of the mark. So, for example, we can safely use “tsinnorit” because it has no correspondingly-named code point.
2. Some Hebrew marks have a correspondingly-named code point, and the mark and its code point are in one-to-one relation:
   1. that is the one and only code point that is used to represent that mark
   2. that is the one and only mark that can be represented by that code point
   In such cases we can safely conflate the name of the mark with the name of its code point. For example, we can safely use “hiriq” to refer to the code point HIRIQ.
3. Finally, we come to the confusing cases. Some Hebrew marks have a correspondingly-named code point, but they are not in one-to-one relation. One or both of the following are true:
   1. more than one code point can be used to represent that mark
   2. more than one mark can be represented by that code point
   In such cases we present the mark’s name using the Hebrew alphabet, and hover-transliterated. The table below shows cases of importance for this document. Marks that can be represented by more than one code point are shown in bordered cells. In two cases this confusion is only present in old Sefaria, and this is marked below.

This document, when viewed in its original HTML form, expects the following fonts to be available on the system:

1. Taamey D
2. Taamey Frank CLM (only a few uses)
3. SBL Hebrew (only a few uses)
4. Ezra SIL (only a few uses)

This document’s default font for pointed Hebrew is a font of my own invention called Taamey D. It is derived from Taamey Frank. It is the only liberally-licensed font that can render all the examples of this document without distracting mark collisions.

Although this document is intended to self-sufficiently describe MAM’s encoding, there are several other documents that may also be useful for readers to consult:

features-of-interest.pdf

This document lists instances of many features of interest. Where feasible, these lists are exhaustive, i.e. every instance of a feature is listed. Many of the features listed there are also covered, from a different perspective, in this document.

README.html & features-of-interest.html

These are the HTML files for this document and for “Features of Interest.” It may be helpful to view them in a text editor or using a browser’s “view source” feature.

The 39 CSV files being documented

These files form the most unfriendly, but of course the most definitive documentation of themselves!

Tag encoding

List of class attribute values

1. mam-spi-{samekh, pe, invnun}
2. mam-implicit-maqaf
3. mam-kq
4. mam-kq-{k, q, trivial}
5. footnote (on <i>)

Legarmeih & פסק

As the table below shows, MAM encodes legarmeih & פסק distinctly, but old Sefaria does not.

The table above shows that MAM distinguishes legarmeih from פסק in the following two ways:

1. Legarmeih uses a SPACE at the end, whereas פסק uses a THIN SPACE.
2. Legarmeih uses <b> whereas פסק uses <small>.

Some examples appear below.

Some further notes on legarmeih vs. פסק:

Although legarmeih & פסק mean quite different things, they are both encoded with PASEQ. They are distinguished by the context around PASEQ.

First, what is in common? Both legarmeih & פסק have a thin space before their PASEQ.

PASEQ-for-legarmeih is followed by a normal-width space, and is styled using whatever the default styling is for the <b> element. The asymmetric spacing around the PASEQ (thin before, normal-width after) indicates that the PASEQ belongs to the word preceding it, not to the word following it. Cleaving to the preceding word indicates that the PASEQ is being used as a legarmeih. Recall that the most common use of legarmeih is to modify the meaning of a munach on the word preceding the legarmeih.

PASEQ-for-פסק is followed by a thin space, and it is styled using whatever the default styling is for the <small> element. In Wikisource MAM, it is both smaller and gray. The symmetric spacing around PASEQ-for-פסק indicates that the mark belongs neither to the word before it, nor to the word after it. Or it belongs to both equally, if you prefer. Recall that פסק serves various functions, all of which somehow separate its neighbor words. It is not an accent, nor does it, like legarmeih, modify the meaning of an accent.

Small, large, & hung letters

MAM uses the following somewhat natural tags for these letters: <small>, <big>, & <sup>. The following table shows example words that illustrate all three types of letters and two exotic contexts: small & large in the same word and small in ketiv.

Old Sefaria stripped special letter encoding from WLC when importing WLC, so there’s no comparable feature.

Samekh, pe, & inverted nun

MAM encloses samekh & pe in curly braces whereas old Sefaria encloses them in parentheses. MAM uses no braces, brackets, or parentheses around inverted nun whereas old Sefaria encloses it in parentheses. MAM encloses these letters in spans with the following classes:

1. mam-spi-samekh
2. mam-spi-pe
3. mam-spi-invnun

These spans include not only the letter itself but also any braces around them. These spans are preceded by   (NO-BREAK SPACE). The span for samekh is followed by eight   in a row. The span for pe is followed by <br>.

The tables below compare the way MAM & old Sefaria use ׆. These examples are complete, i.e. these are the only uses of ׆ in these Tanakh editions. The table below shows, schematically, verses 34, 35, & 36 of Numbers 10. We use an empty square, □, as a placeholder for the first & last words of a verse.

Or, even more schematically:

The table below shows how, in Psalm 107, MAM & old Sefaria agree that:

1. There are two runs of verses separated by ׆.
2. The first run is seven verses long, and thus contains six ׆.
3. The second run is two verses long, and thus contains a single ׆.
4. The second run consists of verses 39 & 40.

The table below also shows how MAM & old Sefaria disagree about the following:

1. Old Sefaria’s run of 7 verses starts at verse 20, two verses earlier than MAM.
2. In old Sefaria, the ׆ separator ends the verse that precedes it whereas in MAM the ׆ starts the verse that follows it.

Implicit maqaf

MAM has 63 maqaf marks that are implied by their accent context but do not exist in medieval manuscripts or most printed editions. They are styled gray in Wikisource MAM. In CSV MAM, they are enclosed in a span with class mam-implicit-maqaf. Here’s an example from Psalm 2:7:

The five main ketiv/qere types

MAM encloses a ketiv/qere pair in a span with class mam-kq. Within this span, the ketiv and qere are enclosed in spans with classes mam-kq-k & mam-kq-q respectively.

For the five main ketiv/qere types in MAM, the table below schematically compares MAM & old Sefaria. We use an empty square, □, as a placeholder for a word preceding a maqaf.

Here is a table with another way to compare ketiv/qere encoding:

Ketiv velo qere & qere velo ketiv

Instances of these two “velo” phenomena are enclosed in spans whose classes are the same as those used for normal ketiv/qere phenomena: mam-kq-k & mam-kq-q respectively. But, unlike instances of normal ketiv/qere, instances of “velo” are not enclosed in a parent span of class mam-kq. As the table below shows, for ketiv velo qere, MAM uses parentheses, whereas old Sefaria does not.

Trivial ketiv/qere pairs

MAM considers 116 kq pairs to be trivial and does not include a ketiv for these pairs. The qere is encoded in a span with class mam-kq-trivial. The table below shows an example from Genesis 13:3.

Notes

There are 29 concise notes in MAM. All of these notes point out variations which appear in Torah scrolls (or scrolls of Esther) that are actually used in synagogues today, i.e. variations that are found in the written letter-text. These notes thus differ substantially from the thousands of textual notes in WLC, which document minor anomalies in vocalization. Old Sefaria stripped notes from WLC when importing WLC, so there’s no comparable feature.

A MAM note is encoded in two parts:

1. an asterisk inside <sup>
2. the note itself, parenthesized inside <i class="footnote">

Here is an example, from Deuteronomy 11:21:

1. <sup>*</sup>
2. <i class="footnote">(בספרי ... גדולה)</i>

Here is what that example looks like, in context:

Observe that the note contains a pointed word with a large letter.

Notes appear after their referent in all cases except one, in Deuteronomy 22:6. There it is better for the callout (the asterisk) to precede rather than follow the word that the note refers to.

Good endings

A “good ending” is an instruction to repeat the second-to-last verse of a book when reading the book publicly. This repetition causes the reading to end on a positive note. By convention, the instruction takes the following somewhat implicit form: the unpointed version of the verse to be repeated appears after the end of the book proper. I.e. there are no instructions per se.

In old Sefaria, good endings appear at the end of three books: Malachi, Lamentations, & Ecclesiastes. In old Sefaria and MAM, a good ending is encoded as follows (the only use of HTML tags in old Sefaria):

<br><small>...</small>

In MAM, good endings appear at the end of four books. Three of these four books are the same three as in old Sefaria. The one additional book is Isaiah. The example below shows the last two verses of Lamentations (5:21 & 22) and the corresponding good ending.

String encoding

Qamats qatan

QAMATS QATAN (QQ) is used in MAM but not in old Sefaria. Unfortunately, in some fonts the distinction between QAMATS (gadol) & QQ is quite subtle. Perhaps counterintuitively, usually QQ is a little taller than QAMATS (gadol). Consider the chanted word וּבְכׇל־הָרֶ֖מֶשׂ in Genesis 1:26, in various fonts:

In texts like MAM, where QQ is used, really the glyph for HATAF QAMATS (HQ) should match the glyph for QQ, because the qamats part of HQ is, conceptually, inherently qatan. But as far as I know, there is no way to accomplish this without a special font. There is no code point for hataf qamats qatan, so a special font would be needed that did one of the following:

1. unconditionally made the glyph for HQ match the glyph for QQ
2. supplied two glyphs for HQ:
   1. a default one, matching QAMATS (gadol).
   2. an alternate one, matching QQ.

(OpenType has a mechanism to provide both a default & an alternate glyph for the same code point. Complementarily, CSS has a mechanism to select such an OpenType alternate glyph.)

The table below shows the word אׇהֳלֹה in Genesis 9:21, in various fonts, allowing easy comparison of HQ & QQ:

As the table below shows, in some fonts, QQ does not mix well with merkha or munach. Fortunately there are only a few such cases in MAM.

אתנח הפוך

The code point ATNAH HAFUKH (AH) is used in MAM but not in old Sefaria.

In both MAM & old Sefaria, the code point YERAH BEN YOMO (YBY) serves both of the following roles:

1. ירח בן יומו (only in the 21 books)
2. galgal (only in Jb, Pr, & Ps)

There is no ambiguity in serving both roles since the roles are exclusive.

In old Sefaria, YBY also serves, ambiguously, as אתנח הפוך, since AH is not used. For example, consider the words אֵין and קִרְבָּם in Psalm 5:10:

גרש-telisha & gershayim-telisha

In five cases in Tanakh, a word is supposed to be notated to indicate that its stressed syllable is to be chanted with first the גרש or gershayim melody and then the telisha gedolah melody. (In the title of this section I have used “telisha” without “gedolah” only for brevity, not for generality, i.e. this phenomenon occurs only with telisha gedolah.)

These five cases present a veritable minefield of problems at various levels.

1. It is not clear what results to aim for, particularly in the two cases where the stressed syllable is not the first syllable.
2. Even where it is clear what results to aim for, some results seem unattainable in some fonts.
3. Old Sefaria is missing gershayim in two cases.

The table below shows, in this document’s default font, all 5 cases as they are encoded in MAM, UXLC, & old Sefaria. (The default font is Taamey D.) Luckily UXLC is always the same as either MAM or old Sefaria, somewhat simplifying this complex situation.

The table below reviews some of the issues with old Sefaria:

Stress זרקא

As far as I know, everyone uses ZINOR for postpositive זרקא.

MAM uses ZARQA rather than ZINOR for stress זרקא. By “stress זרקא” I mean a זרקא that supplements the postpositive זרקא, indicating stress on a nonfinal syllable. Old Sefaria doesn’t (or doesn’t often?) use stress זרקא. For example, consider the word טֶרֶם in Genesis 19:4:

The table below shows the roles that ZINOR & ZARQA play in MAM. These roles are standard, except for MAM’s use of ZARQA for stress זרקא.

No displacement of זרקא by ל

In old Sefaria, in some but not all cases when a word ending in ל needs זרקא, the following things happen:

1. The זרקא is displaced back to the letter before the ל.
2. ZARQA is used for that זרקא, rather than ZINOR.

This stems from the original Michigan-Claremont coding of WLC, where 82L was used for some or all cases in which the LC’s scribe put the זרקא before the ל’s ascender. The code 82 is only documented as representing tsinnorit, but it makes sense that 82 would also be used for this case, which we might call “medial” or “impositive” זרקא.

What doesn’t make sense is why the 82 would precede the L: WLC represents postpositive זרקא after ל as L02, so why wouldn’t medial זרקא on ל be L82? Perhaps L82 at the end of a word was deemed “illegal” due to some seemingly-reasonable rule like the following:

If a word ends in a closed syllable, the only accents that can appear at the end of that word are postpositive.

This rule is reasonable only if the set of postpositive accents is defined reasonably. If adherence to a rule like this is desired, I would recommend Groves do the following:

1. make a distinction between semantically postpositive accents and graphically postpositive accents
2. state the rule in terms of semantically postpositive accents
3. define accent 82 as always graphically impositive, but semantically postpositive in the 21 books, i.e. semantically postpositive when not used in Jb, Pr, & Ps. Or define a new accent code for graphically impositive זרקא.

MAM only uses true postpositive זרקא. The table below compares the handling of the word יִשְׂרָאֵל in Leviticus 4:2 in MAM & old Sefaria. In its third & final row, the table also shows a fix I have proposed for UXLC. With this fix, the זרקא appears on the ל not the א, but still before the ל’s ascender. This proposal will very likely be accepted.

The word יִשְׂרָאֵל appears 9 times like this in old Sefaria. Two words other than יִשְׂרָאֵל appear like this in old Sefaria:

1. אֶל־בְּצַלְאֵ֘ל (Exodus 36:2)
2. תוּכַ֘ל (Deuteronomy 14:24)

Plain old word-final L02 appears 47 times in the WLC. Some of these cases are in Jb, Pr, & Ps, in which case the 02 represents צנור not זרקא. But presumably the same graphical “problem” of the ל’s ascender still applies, whether the mark functions as צנור or זרקא. It would be interesting to know if indeed in all 47 of these cases the scribe put the mark after the ascender. I.e. it would be interesting to know if this scribal choice has been consistently coded in the WLC. There are no cases of 82L past the book of Joshua, suggesting that either the LC’s scribe or the WLC’s transcribers abandoned ל-impositive זרקא at some point.

No special meteg placements

Old Sefaria has two meteg placements that are not present in MAM:

1. early meteg, encoded as METEG before the vowel in question.
2. medial meteg, encoded as the hataf vowel in question followed by ZWJ (ZERO WIDTH JOINER) & then METEG.

Old Sefaria encodes early meteg in a “fragile” manner. This concept is explored at length in the section “Robust Unicode.” For here, suffice it to say that this fragility makes the early meteg look just like a plain (i.e. post-vowel) meteg in most (all?) popular browsers. So, in the table below, we show an example old Sefaria word not only in its actual, fragile form, but also in a somewhat hypothetical robust form, rendered robust through the use of CGJ.

The table below shows examples of these special meteg placements.

Robust Unicode

It would be nice for all texts in the Sefaria system to appear as expected even when subjected to Unicode normalization, since most (all?) popular browsers normalize.

Some words are fragile: in their natural or naive encoding, normalization changes their appearance in at least one font of interest. Fragile words are rare: even though most words in Tanakh have their underlying Unicode changed by normalization, these changes are rarely visible. The most common example of benign normalization has to do with dagesh. Though normalization frequently changes the location of dagesh in the underlying Unicode, I’ve never seen a font in which such changes are visible. Other common examples of benign normalization have to do with shin dot and sin dot.

Though fragile words are rare, it would be nice to encode them robustly. MAM uses CGJ (COMBINING GRAPHEME JOINER) to make fragile words robust to Unicode normalization. A CGJ inserted in the proper place prevents normalization from reordering a string in a way that would change its appearance. The words requiring CGJ come from two sources:

1. words for Jerusalem (this is by far the most plentiful source)
2. words in the three dual cantillation sections of the Tanakh:
   1. Genesis 35:22 (in the Saga of Reuben)
   2. Exodus 20:2–14 (the Exodus Decalogue)
   3. Deuteronomy 5:6–18 (the Deuteronomy Decalogue)

For purposes of making words robust, we consider there to be four types of words for Jerusalem, distinguished by what should happen after the ל:

1. Either patach or qamats gadol can come immediately after the ל.
2. Then comes an accent, in all but one case (Ps147:12). That one accent-less case is re-ordered by normalization but is not fragile so it is not of interest to us here. (I.e. that one accent-less case is re-ordered by normalization, but the normalization is benign, i.e. does not change its appearance. It is a little unclear why this re-ordering does not change its appearance. My best guess is that the shaper (e.g. MS DirectWrite, HarfBuzz, etc.) “undoes” the re-ordering of the vowel marks. Nonetheless in this case we have opted to insert a CGJ before the hiriq.)
3. Then comes one of the following two endings:
   1. hiriq, ם
   2. sheva, מָה. This ending is rare (only four cases).

With a column for each of the four types implied by the variations above, the table below lists:

1. The count of cases of this type in MAM. Not all of these cases are fragile, but it is simplest to treat them as if they were.
2. An example word of this type, shown as it appears in MAM, i.e. shown with CGJ making it robust.
3. This same example word, shown as it appears in old Sefaria, i.e. shown fragile.
4. A location (book, chapter, & verse) containing this example word.

In some environments (font, OS, etc.), the fragile old Sefaria words above don’t just suffer from ordering issues, they also suffer from collision issues (marks overlap). Since no word in Tanakh actually needs the normalized order to be the visual order, it is not surprising (and should not be disappointing) that in some contexts, some Hebrew fonts cannot handle this order. E.g. there is no word in Tanakh that needs the visual order of marks on lamed to be sheva, qamats, etnachta. (This is the normalized order of code points in the example above from 2 Kings 9:28.)

Fragile words in the dual cantillation sections are of two types:

1. The 4 (only 2 unique) “QUPO” words. These are words with a single letter housing four marks:
   1. Qamats gadol
   2. Under-accent (meteg or etnachta)
   3. Patach
   4. Over-accent (revia or גרש)
2. The 10 (only 6 unique) “UM” words. These are words in which an under-accent should be followed by meteg.

Below is a table showing the two unique QUPO words in MAM and their three corresponding words in old Sefaria. Two out of the three old Sefaria words lack a patach on the letter in question, making them QUO not QUPO. On the bright side, QUO is not fragile. More on this further below.

Only the two bordered QUPO words above show a distinction between the appearance of fragile & robust encodings.

The non-bordered words some distracting issues not related to fragility. The table below describes these issues.

Regarding עַל־פָּנָֽ֗יַ in old Sefaria:

1. I reported the “late” patach to the UXLC maintainer and it is now fixed.
2. I’ve shown the sof pasuq (׃) here as if it were in both E & D but old Sefaria accurately reflects that the LC lacks it in E.

The table below shows 3 of the 6 unique UM words, with a row showing U, the under-accent that should be followed by meteg.

Old Sefaria’s Tanakh has no measures to prevent fragility. I have reported the fragility issues to the UXLC maintainer and now most fragile words have been made robust in UXLC. Unfortunately, as far as I know, there is no easy way to update old Sefaria with such a large change (hundreds of changes). The change was made algorithmically to UXLC, so perhaps the easiest way to update old Sefaria would be to run a similar algorithm on its text.

CSV topics

Metadata fields

1. license (not in CSV)
   • MAM: CC-BY-SA 3.0
   • old Sef.: Public Domain
2. versionTitleInHebrew (not in CSV)
   • MAM: מקרא על פי המסורה
   • old Sef.: תנ״ך מלווה בטעמי מקרא
3. versionTitle (Version Title)
   • MAM: Miqra according to the Masorah
   • old Sef.: Tanach with Ta'amei Hamikra
4. versionSource (Version Source)
   • MAM: https://en.wikisource.org/wiki/User:Dovi/Miqra_according_to_the_Masorah
   • old Sef.: http://www.tanach.us/Tanach.xml

(The fields above are shown along with their old Sefaria values, for comparison.) (Some fields are not present in the CSV but they are present in JSON exports from old Sefaria so I figured they may be needed, at some point, in order to import MAM.)

Joshua 21

MAM has no content corresponding to verses 36 & 37 in old Sefaria’s Joshua 21. (The LC also lacks any such content.) In MAM, these verses are encoded as a string containing only an em dash. This makes the CSV for verses 35–38 look like this (in abbreviated form):

(Above, Hebrew strings are shown right-to-left. Also, a slight distraction is that the the Hebrew string for Joshua 21:38 happens to not need quotes.)

If these two verses were not listed at all in the CSV, MAM’s verse numbering would get out of sync with old Sefaria’s for the remainder of Joshua 21. If these two verses were listed as empty strings, another Tanakh version’s content would “interrupt” MAM here. (This is due to the design of the Sefaria system.)

End of document