Skip to content

fix(validators): fold accents in sanitize_str instead of dropping the letter - #21

Merged
TatianaTchouakam merged 1 commit into
devfrom
feat/provinces-accents
Oct 27, 2026
Merged

TatianaTchouakam merged 1 commit into
devfrom
feat/provinces-accents

Conversation

@TatianaTchouakam

Copy link
Copy Markdown
Collaborator

What

sanitize_str removed every character outside a-z0-9, so an accented letter was deleted, not folded:
Takéo -> tako, while the tracker value Takeo -> takeo. They never matched.

Now accents are folded first (é -> e, ư -> u, đ -> d), then spaces and symbols are removed as before:
Takéo, Takeo -> takeo · Nam Định, Namdinh -> namdinh · Hà Nội*, Hanoi -> hanoi.
The official name with accents is still the one published.

The docstrings already said "sanitizing folds accents"; the code now does it.

Tests

  • sanitize_str folds accents (incl. đ).
  • validate_allowed_values accepts a value typed without accents and publishes the canonical one.
  • validate_province with the real list: Takeo, Kratie, Hanoi accepted; unknown name -> Undefined.
  • No two provinces of the list sanitize alike.
  • just ci green.

Effect (local run, --skip-upload --skip-download --skip-drive-download)

before after
province findings 2026 150 146 (−4)
province findings all years 1,170 1,038 (−132)
all findings 104,861 104,729 (−132)

Compared column by column: only province changed.

@pmayd pmayd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, use a squash merge commit :) Just one question, do we store the original province anme or normalize it? So is it Takeo or Takéo? We should probably not have the special chars in the database?

@TatianaTchouakam
TatianaTchouakam merged commit cbc24b6 into dev Oct 27, 2026
2 checks passed
@TatianaTchouakam
TatianaTchouakam deleted the feat/provinces-accents branch October 27, 2026 15:48
@TatianaTchouakam

Copy link
Copy Markdown
Collaborator Author

Thanks! We store the official name from allowed_provinces.yaml, so “Takéo” with the accent. The version without accents is only used to compare, it is never published. This did not change with this PR: before, the official name with accents was already published; now “Takeo” written without accent is also recognised.
About special characters in the database: the official names come from the master list, and for Vietnamese the accents are part of the name. If accents are a problem for joins or maps in Looker, we could add a second column without accents (in the pipeline or in a BigQuery view) and keep the official name. Happy to do it if you want.

@pmayd

pmayd commented Oct 28, 2026 via email

Copy link
Copy Markdown
Collaborator

@TatianaTchouakam

TatianaTchouakam commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks! Done in #30: a second column with the province name without accents (e.g. “Takéo” → “Takeo”, “Hà Nội” → “Ha Noi”), in the clinic master data (clinic_province_ascii) and in the patient static table (province_ascii). The official name stays as it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants