Add stemmer for Tagalog

[ikalchev]

This attempts to resolve the Tagalog part of #273 .

Disclosure: LLMs heavily influenced the rewrite. I tried to validate the end result for sanity to the best of my snowball knowledge. Feedback and suggestions are more than welcome.

[ojwb]

This attempts to resolve the Tagalog part of #273 .

Disclosure: LLMs heavily influenced the rewrite.

Dear me, their grasp of Snowball seems really poor...

[ojwb]

tomark can only advance the cursor, not move it back (in forwards mode that is - the opposite is true in backwards mode) so using setmark start and tomark start like this to reset the cursor to the start of the word just can't work.

However it's also unnecessary since do restores the cursor to where it was before the command - it looks to me like the cursor will always be at the start whenever you're doing tomark start (in fact that probably has to be true since tomark start will signal f if the cursor is already after start).

[ojwb]

Tests like these are bad because the cursor counts in bytes for UTF-8 but in Unicode codepoints otherwise, so limit - cursor depends on the encoding in use.

It looks like Tagalog words only use ASCII so the impact is significantly reduced, but it means foreign proper nouns can stem differently, which is unhelpful and so we try to avoid writing tests like the above - you can try to move the cursor instead:

test hop 3

If the restriction is global (or applies to a section of the stemmer), it's better to use setlimit once up-front instead of checking the remaining stem everywhere.

[ojwb]

Another encoding-dependent check - the portable version is:

$(size >= 5)

But again, using setlimit may be better.

[ojwb]

It would be better to combine the amongs from all the numbered subfunctions into a single big among here. An among picks the longest matching substring, so there's no need to split the strings up by decreasing length like this, and a large among is more efficient (currently among is O(log(n)) in the number of strings; I'm working on a new implementation which is O(1)).

[ojwb]

It's weird to test two single character strings in a row like this - more natural is to write 'ng' instead of 'g' 'n' (order swaps because this is in backwardmode, but that makes it clearer that this actually matches the suffix -ng).

[ojwb]

These two routines are only used in special_cases or exception1 and are both [substring] atlimit among (...) - it'd be more efficient to merge them into a single routine with a single among.

I'd argue it'd be clearer code too - they're really both handling exceptional cases to the stemming process.

(It also seems odd to have exception1 but not any other routines named exception.)

[ojwb]

It looks like the two cases above could be more cleanly handled by this among - something like (untested):

[substring] among (
     'tong' 
         ( has_minimum_suffix_stem <-'ton' set removed_suffix )
     'iong' 
         ( has_minimum_suffix_stem <-'ion' set removed_suffix )
     'yong' 
         ( has_minimum_suffix_stem <-'yon' set removed_suffix )
     'ng' (
         $(len >= 7)
         test vowel
         has_minimum_suffix_stem
         delete
         set removed_suffix
     )
     // Other cases as below...

[ojwb]

These 3 dos would be better done using among.

[ojwb]

It looks to me like you only need a single removed flag since these are only ever tested as:

(removed_prefix or removed_infix or removed_duplication or removed_suffix)

or:

(removed_prefix or removed_infix or removed_duplication)

(and the latter test is only used before removed_suffix can be set).

[ojwb]

You need to also add an entry to libstemmer/modules.txt - otherwise the tests won't actually test it. Something like:

tagalog         UTF_8,ISO_8859_1        tagalog,tl,tgl,fil

https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes has "Filipino (Pilipino) has the code fil" in the "Notes" column for Tagalog. Wikipedia says "[Filipino] is a de facto standardized form of the Tagalog language", so I've included fil in the aliases based on that. Perhaps we should also include filipino as an alias. It'd be useful to have input from someone familiar with the relationship between them.

[ikalchev]

I much appreciate the review, thank you. Is the new version better? There are slight differences to the stemmer output (I've updated the data PR accordingly).

[ojwb]

There are slight differences to the stemmer output

Hmm, I wonder why - the changes I suggested shouldn't really change output for ASCII inputs. I'll have a look.

Do the stems seem better before or after?

[ojwb]

I've tried applying each change separately, and the change in output is entirely from merging into a single among in remote_prefix.

The issue is that the longest matching suffix is chosen and then the rule for it is run, so for example, a word ending -naka would take the 'naka' case - then if has_minimum_remainder fails, nothing is done. Before we would cascade to shorter rules, and consider the 'ka' case.

This could be addressed by using has_minimum_remainder as an "among function" (the among then picks the longest suffix for which the among function succeeds), but it'd be better to apply the has_minimum_remainder requirement by setting the limit to be at least 3 characters in and at least two vowels in. The wrinkle is that the less strict has_replacement_remainder is used for one case...

However a key question is whether the old or new behaviour is actually better. A similar situation occurs in other languages, and there's no single right answer - sometimes one approach works better and sometimes the other does.

I ran the stemmer-compare script for the change:

compare.html

Firefox's machine translation doesn't support Tagalog or Filipino, but cutting and pasting chunks into google translate suggests it's probably worse overall with the change, but there definitely appear to be some improvements.

For example, before tagalog gets overstemmed to alog, as does magtagalog, but pagtataglog stems to tagalog. After the change these all stem to tagalog.

[ojwb]

I really don't understand this not - there seem to be several problems here.

If we get here, then $(size >= 5) and the string ends ...vowel'in'non-vowel and the cursor will be right before 'in' (because we test vowel so the cursor isn't advanced over the vowel). That means the first atlimit is always false, because we know there's a vowel (and also at least another character because of $(size >= 5)) before the limit.

Ignoring that, if the first atlimit is true then 'd' can't be.

Also goto ('i' 'y' 'a' or 'i' 't' 'a') atlimit makes little sense - goto advances the cursor until the command inside succeeds, so if the goto succeeds the cursor will be at the start of the matched sequence and so the second atlimit won't ever be true.

I'm not sure what's intended for the pattern inside either. It's peculiar to test multiple single character strings in succession like this. The or binds tightly, so it actually matches either iyata or iyita.

Is the intent here is to check if the word has -iyata or -iyita? If so, then we can more clearly and more efficiently check that with:

backwards among ('iyata' 'iyita')

(The has_minimum_infix_remainder seems redundant here - there will always be 2 characters after the cursor because $(size >= 5), and if there aren't two vowels then the -iyata/-iyita suffix check will fail anyway.)

[ojwb]

Is the intent here is to check if the word has -iyata or -iyita?

I don't think it can be as no words in tagalog/voc.txt contain either of those sequences, even without anchoring to the end of the word:

$ grep iyata tagalog/voc.txt 
$ grep iyita tagalog/voc.txt 
$ 

So I'm guessing the intent was actually to match either -iya or -ita, which occur as a suffix 112 and 102 times respectively. If so that can be tested with:

backwards among ('iya' 'ita')

[ojwb]

This seems a plausible first draft, but has some significant problems which I think are going to take a while to resolve, so I've put it on the 3.2.0 milestone. If it turns out to be ready to merge before 3.1.0, we can reconsider but I wouldn't delay 3.1.0 for it.

[ikalchev]

I made another attempt at it but this time I am really not sure if it is better than the prev one.

[ojwb]

I made another attempt at it but this time I am really not sure if it is better than the prev one.

I looked at the stemmer this is apparently based on, and the results of that seem surprisingly different to this:

A total of 7198 words changed stem
1197 words changed stem but aren't interesting
298 merges of groups of stems: 
[...]
825 splits of groups of stems:
[...]
3179 words moving between stem groups:
[...]

That makes me wonder if we're heading down a dead-end here and if we'd be better off throwing away the LLM-generated Snowball code and just starting again from the existing stemmer. Coding up a stemmer in Snowball isn't really the hard part of this.

(The stats from https://github.com/andrianllmm/tagalog-stemmer#accuracy seem perhaps a little underwhelming, but I'm not sure what equivalent numbers for our existing stemmers would actually look like.)

[ikalchev]

@ojwb Note that that stemmer uses a valid word list to improve accuracy and for disambiguation.

Also it tries to handle infixes and I am not sure how that plays with snowball.

That makes me wonder if we're heading down a dead-end here and if we'd be better off throwing away the LLM-generated Snowball code

I trust your judgement on this. At least from what we've seen internally so far, it does provide some value that is better than no stem at all. We are trying to get more validation.

[ojwb]

Note that that stemmer uses a valid word list to improve accuracy and for disambiguation.

Yes, I've seen that.

You can turn it off by passing None as a second parameter to get_stem(). A quick peer at the results with/without suggests it's worse without (which isn't a total surprise). We wouldn't want a 40000+ word list in a Snowball stemmer, but my guess is most of the benefits can be got with a much smaller list of exceptional cases.

Also it tries to handle infixes and I am not sure how that plays with snowball.

You can remove infixes using Snowball (e.g. the Dutch algorithm removes -ge-). They do tend to be harder to remove without unwanted consequences though, probably mainly because there's just more scope for the infix's letter sequence to appear in words where it's not actually the infix. That's why we don't try to remove them in German currently (#223): removing -ge- in German seems very hard to do reliably; removing -zu- seems possible but needs a fairly large list of prefixes which can occur before it, and some exceptions.

That makes me wonder if we're heading down a dead-end here and if we'd be better off throwing away the LLM-generated Snowball code

I trust your judgement on this. At least from what we've seen internally so far, it does provide some value that is better than no stem at all. We are trying to get more validation.

Are you a Tagalog/Filipino speaker? Or have one involved?

[kei-nan]

Hey @ojwb,
I don't think @ikalchev will work on the PR.
It was opened as part of his work with us; but he moved on to a different company.
I don't have access to his fork, so not sure how we could further modify the branch.
Let me know how you want to proceed.

[ojwb]

You could fork his fork and open a new PR, but my evaluation of this PR compared to the stemmer it's supposed to be based on leads me to suspect this PR is probably not a good starting point (see comments above).

From past experience, the hard part of making a new Snowball stemmer is not implementing the algorithm in Snowball, but rather actually designing the algorithm. Resolving cases of overstemming can be especially tricky, though it varies a lot by language; omitting problematic rules solves overstemming, but can create many cases of understemming instead; adding a large number of exceptions to rules isn't ideal, especially as it can miss words which aren't in the vocabulary we evaluate with (also neologisms). You could think of it as like over-fitting.

ikalchev found an existing Tagalog stemmer, so the questions I'd try to answer first are whether that stemmer actually does a good job of what we want it to do [*], and whether it still does a good job without its dictionary. If the dictionary is really needed, can we adapt it to use a much smaller list of exceptions instead?

[*] It describes itself as "a library that finds the root form of Tagalog words", which is close to but not exactly what we're trying to do here. Snowball is intended for improving recall in Information Retrieval, and for that purpose what matters is that forms of a word with a sufficiently similar meaning are conflated onto a common stem, but also (and especially important) that words with different meanings are mapped to different stems. We don't directly care if the stem is the linguistic root or not (it often is in practice, but sometimes it's not exactly the root form - e.g. the English stemmer maps signify to signifi, but it does the same to signifier and that's OK). We will also generally prefer understemming to overstemming if we have to choose, whereas if the aim is to return the root form you might chose different trade-offs. Finally, the linguistic root form may have a different enough meaning that conflation in search is unhelpful (e.g. awful in modern English means "very bad", but its linguistic root form is awe which means "a feeling of fear and reverence"/"a feeling of amazement").

Do you have someone reasonably fluent in Tagalog/Filipino involved? I asked before but I don't see an answer.

Incidentally if you have a budget for this work, I'm hireable on a freelance basis. It's certainly not a prerequisite for getting code merged, but is likely to be more efficient than giving the task to someone unfamiliar with Snowball.