Conversation
Add a Substring value to SearchType that matches the query anywhere within a string, case- and diacritic-insensitive, backed by a raw-string index scanned with CompareInfo.IndexOf. The existing Exact/Prefix/FullText modes are unchanged.
| return Enumerable.Empty<T>(); | ||
| CompareInfo ci = CultureInfo.InvariantCulture.CompareInfo; | ||
| return raw.Where(kv => ci.IndexOf(kv.Value, text, | ||
| CompareOptions.IgnoreCase | CompareOptions.IgnoreNonSpace) >= 0).Select(kv => kv.Key); |
There was a problem hiding this comment.
we might need to consider if this will work correctly for us, or if we need to create a manged wrapper in icu-dotnet for string search so we can use what ICU provides for this use case.
hahn-kev
left a comment
hahn-kev
left a comment
There was a problem hiding this comment.
This looks good. I left some suggestions, nothing blocking.
One thing, since the issue specifically calls out matching how FW Lite search works, here's some of our tests which would be relevant here
https://github.com/sillsdev/languageforge-lexbox/blob/bbcc058f267a789302b9570e9542d30968199a3f/backend/FwLite/MiniLcm.Tests/QueryEntryTestsBase.cs#L412-L455
Up to you how closely you try to match what FW Lite does.
| #endregion SortKeyIndex class | ||
|
|
||
| private readonly Dictionary<Tuple<int, int>, SortKeyIndex> m_indices = new Dictionary<Tuple<int, int>, SortKeyIndex>(); | ||
| private readonly Dictionary<Tuple<int, int>, List<KeyValuePair<T, string>>> m_rawIndices = new Dictionary<Tuple<int, int>, List<KeyValuePair<T, string>>>(); |
There was a problem hiding this comment.
I wish I had a suggestion, but this type is really ugly lol. I might replace KeyValuePair with a custom struct which does the same thing but is more understandable what's going on when you look at it.
Another idea (bigger tweak) is to create a custom struct (maybe named SearchBucket or IndexKey) which is a new struct that replaces all the uses of Tuple<int, int> (which is quite meaning less IMO), you would just need to match the compare and equality rules of the Tuple. This more falls in line with leaving code better than when you found it.
| /// </summary> | ||
| public void Add(T item, int indexId, int wsId, string text) | ||
| { | ||
| SortKeyIndex index = GetIndex(indexId, wsId); |
There was a problem hiding this comment.
In the new SearchType we're now getting the index and never using it, it also is creating the index if it doesn't exist which isn't great since it's never used. The same issue exists on the Search method.
| break; | ||
|
|
||
| case SearchType.Substring: | ||
| GetRawIndex(indexId, wsId).Add(new KeyValuePair<T, string>(item, text ?? string.Empty)); |
There was a problem hiding this comment.
we're handling the case of a null text argument here by using an empty string, but searching for empty strings is explicitly banned. Should we just not add a value to the index if the text is null or empty? We could probably safely do this on the first line of this method. Taking a quick look at the EntryGoSearchEngine and others it looks like they never call Add with an empty string so that's probably safe to change.
2 participants
Add a Substring value to SearchType that matches the query anywhere within a string, case- and diacritic-insensitive, backed by a raw-string index scanned with CompareInfo.IndexOf. The existing Exact/Prefix/FullText modes are unchanged. This change enables a change in FieldWorks to use this new SearchType, see sillsdev/FieldWorks#1069
https://jira.sil.org/browse/LT-22524
This change is