Skip to content

Stop claiming Article/Video have one source each - #742

Open
mircealungu wants to merge 1 commit into
masterfrom
wt/source-id-truth
Open

mircealungu wants to merge 1 commit into
masterfrom
wt/source-id-truth

Conversation

@mircealungu

Copy link
Copy Markdown
Member

Follow-up to #741. That PR declared the indexes production has and the models lacked; this is the same comparison run backwards — constraints the models declare that production does not have, so production already holds rows the test database would refuse.

Article.source_id was never one-to-one

Source.find_or_create deduplicates by (content, type, language) — it returns an existing source when the content matches (source.py:66). Two articles with identical content therefore share a source_id by design: an original and a simplification that came out identical, or the same piece crawled twice.

So unique=True contradicted the very function that populates the column. Production has never had the index and currently holds 6465 source_ids shared by more than one article.

CLAUDE.md went further and documented the relationship as "enforced by unique constraint on source_id" — not true in either direction. Corrected, with a warning about the Bookmark.source_id == Article.source_id joins, which fan out wherever a source has several articles.

Video.source_id was costing data

Same false constraint, but here it had a victim. The crawl deliberately threw away the source link for every broken video:

# TODO: Remove this temporary workaround (this is because source_id is unique in video table)
if video_info["broken"] != 0:
    source = None

Broken videos often have empty or identical text, so several resolve to the same Source and would have collided. With the index gone they keep their source like any other video.

This is the one behaviour change in the PR — worth running a video crawl before merging. It is also why video.source_id shows 0 duplicates in production: the workaround was suppressing them, not the data being naturally unique. (I had initially suggested adding UNIQUE(source_id) to video on the strength of that 0; checking the code first is what caught it.)

Article.img_url_id

Stale in the plainest way: the index was dropped in production by tools/migrations/24-03-26-drop_unique_constraint_from_img_url_id.sql and the model was never updated. Two articles may obviously share an image.

Testing

DEV_SKIP_TRANSLATION=1 python -m pytest zeeguu/core/test zeeguu/api/test -q

498 passed, 12 subtests — unchanged from this branch point.

Scope

Code-only, no migration: production already lacks all three indexes, so the models are moving toward production, not away from it.

Still outstanding from the backwards sweep, pending a duplicate check on production data: article_topic_user_feedback, level_adapted_article_summary_context, phrase, search_filter, search_subscription, topic_filter, topic_subscription. Each needs to be either added to production or dropped from the model, and the data decides which.

🤖 Generated with Claude Code

Source.find_or_create deduplicates by (content, type, language): it returns
an EXISTING source when the content matches. Two articles with identical
content therefore share a source_id by design -- an original and a
simplification that came out identical, or the same piece crawled twice.

So the unique=True on Article.source_id contradicted the very function that
populates it. Production has never had the index and currently holds 6465
source_ids shared by more than one article; declaring it on the model only
made the test database refuse rows production accepts every day. CLAUDE.md
went further and documented the one-to-one as "enforced by unique constraint
on source_id", which was not true in either direction.

Video.source_id had the same false constraint, and it was costing data: the
crawl deliberately dropped the source link for every broken video

    # TODO: Remove this temporary workaround (this is because source_id is unique...)
    if video_info["broken"] != 0:
        source = None

because broken videos often have empty or identical text and so resolve to
the same Source. With the index gone they keep their source like any other
video. That is the one behaviour change here.

Article.img_url_id was stale in the same way: the index was dropped in
production by 24-03-26-drop_unique_constraint_from_img_url_id.sql and the
model was never updated.

Found by comparing the live production index list against db.metadata, the
same sweep as #741 -- this is that comparison run backwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ArchLens - No architecturally relevant changes to the existing views

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant