Skip to content

Remove fragments from Yoruba stop words (#14036) - #14037

Open
adedejimakinde wants to merge 1 commit into
explosion:masterfrom
adedejimakinde:fix-yo-stop-words
Open

adedejimakinde wants to merge 1 commit into
explosion:masterfrom
adedejimakinde:fix-yo-stop-words

Conversation

@adedejimakinde

Copy link
Copy Markdown

Description

Fixes #14036.

The Yorùbá stop word list contained 24 entries that are not standalone words: three bare combining marks (U+0300, U+0301, U+0323), fourteen isolated consonants (b, d, f, g, j, k, l, p, r, s, t, w, y, ṣ), and seven vowels that only occur inside words (à, e, è, i, ì, u, ù). These look like the result of splitting text that wasn't Unicode normalised.

This PR removes them, taking the list from 91 to 67 entries. Single-character entries that are real Yorùbá words are kept, for example ó (he/she/it), ń (progressive marker), ò (negation), and the object pronouns á, é, í, ú.

It also adds spacy/tests/lang/yo/test_stop_words.py, which checks that every stop word is NFC normalised, that no entry starts with a combining mark, that the removed fragments stay out, and that real single-character words stay in.

I ran the new tests against spaCy 3.8.16 before and after the change: 25 failed and 5 passed with the original list, and all 30 passed with the fixed list.

Types of change

Bug fix (language data)

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the new Yorùbá tests against spaCy 3.8.16 (all pass); the full suite will run in CI.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Yorùbá stop word list contains bare diacritics and consonant fragments

1 participant