Stop claiming Article/Video have one source each - #742
Open
mircealungu wants to merge 1 commit into
Open
mircealungu wants to merge 1 commit into
mircealungu wants to merge 1 commit into
Conversation
Source.find_or_create deduplicates by (content, type, language): it returns
an EXISTING source when the content matches. Two articles with identical
content therefore share a source_id by design -- an original and a
simplification that came out identical, or the same piece crawled twice.
So the unique=True on Article.source_id contradicted the very function that
populates it. Production has never had the index and currently holds 6465
source_ids shared by more than one article; declaring it on the model only
made the test database refuse rows production accepts every day. CLAUDE.md
went further and documented the one-to-one as "enforced by unique constraint
on source_id", which was not true in either direction.
Video.source_id had the same false constraint, and it was costing data: the
crawl deliberately dropped the source link for every broken video
# TODO: Remove this temporary workaround (this is because source_id is unique...)
if video_info["broken"] != 0:
source = None
because broken videos often have empty or identical text and so resolve to
the same Source. With the index gone they keep their source like any other
video. That is the one behaviour change here.
Article.img_url_id was stale in the same way: the index was dropped in
production by 24-03-26-drop_unique_constraint_from_img_url_id.sql and the
model was never updated.
Found by comparing the live production index list against db.metadata, the
same sweep as #741 -- this is that comparison run backwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
ArchLens - No architecturally relevant changes to the existing views |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #741. That PR declared the indexes production has and the models lacked; this is the same comparison run backwards — constraints the models declare that production does not have, so production already holds rows the test database would refuse.
Article.source_idwas never one-to-oneSource.find_or_creatededuplicates by(content, type, language)— it returns an existing source when the content matches (source.py:66). Two articles with identical content therefore share asource_idby design: an original and a simplification that came out identical, or the same piece crawled twice.So
unique=Truecontradicted the very function that populates the column. Production has never had the index and currently holds 6465 source_ids shared by more than one article.CLAUDE.mdwent further and documented the relationship as "enforced by unique constraint on source_id" — not true in either direction. Corrected, with a warning about theBookmark.source_id == Article.source_idjoins, which fan out wherever a source has several articles.Video.source_idwas costing dataSame false constraint, but here it had a victim. The crawl deliberately threw away the source link for every broken video:
Broken videos often have empty or identical text, so several resolve to the same
Sourceand would have collided. With the index gone they keep their source like any other video.This is the one behaviour change in the PR — worth running a video crawl before merging. It is also why
video.source_idshows 0 duplicates in production: the workaround was suppressing them, not the data being naturally unique. (I had initially suggested addingUNIQUE(source_id)tovideoon the strength of that 0; checking the code first is what caught it.)Article.img_url_idStale in the plainest way: the index was dropped in production by
tools/migrations/24-03-26-drop_unique_constraint_from_img_url_id.sqland the model was never updated. Two articles may obviously share an image.Testing
498 passed, 12 subtests — unchanged from this branch point.
Scope
Code-only, no migration: production already lacks all three indexes, so the models are moving toward production, not away from it.
Still outstanding from the backwards sweep, pending a duplicate check on production data:
article_topic_user_feedback,level_adapted_article_summary_context,phrase,search_filter,search_subscription,topic_filter,topic_subscription. Each needs to be either added to production or dropped from the model, and the data decides which.🤖 Generated with Claude Code