Skip to content

Add T-value option for collocations - #2142

Open
lukavdplas wants to merge 25 commits into
developfrom
feature/collocation-t-value
Open

Add T-value option for collocations#2142
lukavdplas wants to merge 25 commits into
developfrom
feature/collocation-t-value

Conversation

@lukavdplas

@lukavdplas lukavdplas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Adds a new option for calculating collocation scores based on the T-value. Also significantly expands the documentation for the neighbouring words graph.

T-values seem to work in providing a more "balanced" score, compared to the old formula. Results seem reasonable to me but I'll let the people & parliament team test it after release.

Some comments:

  • An issue with these formulas is that we have to take a lot of shortcuts with the statistics (now also documented in the manual), e.g. taking background frequencies from the entire corpus instead of the query set. So the validity is... questionable.
  • The t-value formula is also available in n-grams mode, but whether that is really valid, I don't know.

@lukavdplas lukavdplas linked an issue Aug 20, 2026 that may be closed by this pull request
@lukavdplas
lukavdplas marked this pull request as ready for review August 20, 2026 10:40
@lukavdplas lukavdplas added visualisation changes to visualisation features next release Issue or pull request should be resolved before next release frontend changes to the angular frontend labels Aug 25, 2026
@lukavdplas lukavdplas added the backend changes to the django backend label Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend changes to the django backend frontend changes to the angular frontend next release Issue or pull request should be resolved before next release visualisation changes to visualisation features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ngrams: new "compensate for frequency" formula

1 participant