Repository navigation
fix(validators): fold accents in sanitize_str instead of dropping the letter - #21
Conversation
pmayd
left a comment
There was a problem hiding this comment.
LGTM, use a squash merge commit :) Just one question, do we store the original province anme or normalize it? So is it Takeo or Takéo? We should probably not have the special chars in the database?
|
Thanks! We store the official name from allowed_provinces.yaml, so “Takéo” with the accent. The version without accents is only used to compare, it is never published. This did not change with this PR: before, the official name with accents was already published; now “Takeo” written without accent is also recognised. |
|
That isnl a very good idea and we should do it I don't think big query has problems with utf characters but still it might become a problem at a later point in time and it never hurts to have a sanitized column in addition so you could add this to the pipeline as additional step when reading the clinic master data
Sent from [Proton Mail](https://proton.me/mail/home) for Android.
…-------- Original Message --------
On Wednesday, 10/28/26 at 09:13 Tatiana Tchouakam ***@***.***> wrote:
TatianaTchouakam left a comment [(CorrelAid/a4d#21)](#21 (comment))
Thanks! We store the official name from allowed_provinces.yaml, so “Takéo” with the accent. The version without accents is only used to compare, it is never published. This did not change with this PR: before, the official name with accents was already published; now “Takeo” written without accent is also recognised.
About special characters in the database: the official names come from the master list, and for Vietnamese the accents are part of the name. If accents are a problem for joins or maps in Looker, we could add a second column without accents (in the pipeline or in a BigQuery view) and keep the official name. Happy to do it if you want.
—
Reply to this email directly, [view it on GitHub](#21?email_source=notifications&email_token=ACJLHU2XFJTV3KFDWO43WVL5WGTC7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMNBWGQYDEOJSGQY2M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6464029241), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ACJLHU3RNH7CIWKHYBZS2KL5WGTC7AVCNFSNUABFKJSXA33TNF2G64TZHM4TINBTGQ4DKMRXHNEXG43VMU5TKOJYGEZTANZVGI32C5QC).
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Thanks! Done in #30: a second column with the province name without accents (e.g. “Takéo” → “Takeo”, “Hà Nội” → “Ha Noi”), in the clinic master data (clinic_province_ascii) and in the patient static table (province_ascii). The official name stays as it is. |
What
sanitize_strremoved every character outside a-z0-9, so an accented letter was deleted, not folded:Takéo->tako, while the tracker valueTakeo->takeo. They never matched.Now accents are folded first (é -> e, ư -> u, đ -> d), then spaces and symbols are removed as before:
Takéo,Takeo->takeo·Nam Định,Namdinh->namdinh·Hà Nội*,Hanoi->hanoi.The official name with accents is still the one published.
The docstrings already said "sanitizing folds accents"; the code now does it.
Tests
sanitize_strfolds accents (incl.đ).validate_allowed_valuesaccepts a value typed without accents and publishes the canonical one.validate_provincewith the real list:Takeo,Kratie,Hanoiaccepted; unknown name ->Undefined.just cigreen.Effect (local run,
--skip-upload --skip-download --skip-drive-download)Compared column by column: only
provincechanged.