tamil: accusative -னை preserves stem consonant ன்

[ithiru]

Bug

The rule '{nnna}{vs_ai}' (delete) at algorithms/tamil.sbl:279 (inside remove_vetrumai_urupukal — "remove case suffixes") deleted both the consonant ன (U+0BA9) and the vowel-sign ை (U+0BC8) when stripping the Tamil accusative case suffix -னை. This left -ன்-ending masculine nouns with a truncated stem that did not match their nominative form, breaking inflection-class collapse across the corpus.

input	upstream	fixed
கர்ணன் (nom)	கர்ணன்	கர்ணன்
கர்ணனை (acc)	கர்ண ❌	கர்ணன் ✓
கர்ணனால் (instr)	கர்ணன்	கர்ணன்
கர்ணனின் (gen)	கர்ணன்	கர்ணன்
பீமனை (acc)	பீம ❌	பீமன் ✓
ஃபவுண்டேஷனை (acc)	ஃபவுண்டேஷ ❌	ஃபவுண்டேஷன் ✓
ஹோர்மோனை (acc)	ஹோர்மோ ❌	ஹோர்மோன் ✓

-ர்-ending masculine nouns are not affected: their accusative is -ரை (handled by a different rule that already preserves -ர்). Pronouns shorter than 5 characters (அவன் / அவனை) are not touched because has_min_length requires len > 4.

Fix

Change the rule body from delete to <- '{nnna}{pulli}': instead of deleting the bigram னை, replace it with ன் (consonant + pulli), restoring the canonical nominative form. One-line behavioural change.

Cross-validation against snowball-data

Stemmed the full snowball-data/tamil/voc.txt (443,271 words) with both upstream and patched algorithms. Diff:

• 2,785 entries change (1.25% of the corpus)
• 1,283 gain a final ன்/ண் consonant that belongs to the noun stem (-னை directly stripped — the bug). New stems are longer.
• 1,329 reach a deeper dictionary root because the corrected mid-step now allows downstream rules to fire that previously couldn't (e.g. அதிகாரத்தினை: அதிகாரத்தி -> அதிகாரம், the dictionary form for "authority"). New stems are shorter but more correct.
• 173 end at the same length but a different consonant.

Manual inspection across each bucket: all changes are corrections, none introduce regressions. The companion data PR (snowball-data#fix/tamil-accusative-nai) regenerates tamil/output.txt so CI passes against this branch.

History

This rule has been unchanged since at least the September 2023 optimization pass. The bug is upstream of all consumers (Python snowballstemmer, the Go blevesearch/snowballstem port, etc.). Tested with snowball master at HEAD before applying the patch.

[ojwb]

Thanks for raising this.

I ran the change through scripts/stemmer-compare in snowball-data, which shows the effects of a change in terms of how the grouping of words by stem changes - here's the HTML report (the graphs at the bottom show existing on the left, proposed on the right):

compare.html

I can't read Tamil, but tried machine translation which can often give a good enough picture. Unhelpfully Firefox's translation says it supports Tamil in the docs and I can download offline translation data for Tamil, but if I try to translate a page it says translation from Tamil isn't supported. Therefore I've only looked at it with Google's translation, which doesn't seem to translate text in embedded SVG images which is a large part of this particular report.

Based on this, it looks to me like:

• the 433 merges seem to be overwhelmingly improvements
• the 120 splits seem to be mostly unhelpful
• the 2948 words moving between stem groups I can't easily tell (I tried copy and pasting a few words to translate them - e.g. the last group seems an improvement as it splits "hydroscopic" and "hydrophone", and instead groups two words translating to "hydrophone"; it's too laborious to translate enough samples like this to really get an accurate picture though)

From what I can see, I wonder if there's an additional change which would help this one - either a condition on when we do this, or (I suspect more likely) adjusting another rule to avoid many of the splits (and probably to avoid unhelpful moves between groups). Can you have a look through the attached compare.html and see what you think?

[ojwb]

Related to evaluating this, the current word list we have for tamil seems too large. Looking at the history, the original word list was 433,918 words extracted from tamil wordnet. It seems this was just all of the words, so will include a lot of very obscure and rare words and forms of words, whereas we really want a representative sample of words that the stemmer will actually encounter in practice.

For licensing reasons, instead of the wordnet list we extracted a list from a 2018 dump of Tamil wikipedia - I wrote in snowballstem/snowball-data#1 :

So I've added a Tamil wordlist extracted from a Tamil Wikipedia dump, along with the simple script used to generate it (which essentially just collates matches for the regexp \b[\x{b80}-\x{bff}]+\b and then outputs any which occur 3 or more times). This contains 443271 unique words.

I didn't record why I chose a threshold of >= 3 occurrences, but I strongly suspect I was just trying to match the same approximate number of entries (since 443271 is close to 433918).

However the original 433918 likely includes obscure words and rare forms of words which didn't appear 3 times in wikipedia, so selecting a similar number of words from wikipedia based on frequency probably included a significant proportion of typos and other junk non-words. We've extracted wordlists for some other languages from wikipedia dumps, and experience from that suggests to me that a threshold of 3 won't work well - it's much lower than we've typically used: it looks like the lowest threshold otherwise was 7 for Indonesian (which had a relatively low number of pages on wikipedia), but otherwise it's at least 10.

Aside from Arabic (where the current wordlist was created mechanically by applying grammar rules to root words which produced an overly complete list which we really need to replace), the current Tamil vocabulary is ~2.4 times larger than the next largest and more than 8 times larger than the average.

Perhaps we should sort out a smaller, cleaner wordlist first as junk in the current one may be obscuring the results. I fetched the latest Tamil wikipedia dump and it's grown significantly since 2018, which should mean we can now get a better list of words of a given size. I'll try processing it further.

[ojwb]

@ithiru Ping - I wonder if you missed my comments and questions above as it's been over a month without a response.