Skip to content

fix: support large input texts with chunking - #100

Open
RSKKSOFFICIAL wants to merge 3 commits into
nextcloud:mainfrom
RSKKSOFFICIAL:fix/chunk-long-text
Open

fix: support large input texts with chunking #100
RSKKSOFFICIAL wants to merge 3 commits into
nextcloud:mainfrom
RSKKSOFFICIAL:fix/chunk-long-text

Conversation

@RSKKSOFFICIAL

Copy link
Copy Markdown

Fixes #71

What this does

Large input texts (>250 words) were being silently truncated by the model
because the output hit the decoding length limit mid-document. This adds a
chunking layer that splits the input into smaller pieces before translation,
then joins the results back into a single output.

Changes

lib/Service.py

  • Added _chunk_text(): splits input at sentence boundaries into chunks of
    max 80 words. Hard-splits any single sentence that exceeds the limit.
  • Added _join_chunks(): joins translated chunks in document order. Uses an
    empty string separator for no-space languages (zh, ja, th, etc.) and a
    single space for all others. Chunk order is always preserved regardless of
    source/target script direction — each chunk is already translated correctly
    by the model independently.
  • Updated translate(): applies chunking when input exceeds the threshold,
    caps max_decoding_length proportionally per chunk to prevent runaway
    repetition loops, and enforces a minimum repetition_penalty per chunk.

config.json

  • Added chunking section with four configurable parameters:
    • chunk_threshold (250): word count above which input is chunked
    • chunk_size (80): max words per chunk
    • min_repetition_penalty (1.5): lower bound for repetition penalty per chunk, prevents output loops on dense scripts like Devanagari
    • max_decoding_multiplier (3): output token cap as a multiple of input tokens per chunk

Testing

Tested with 350+ word inputs across 7 language pairs:

  • English → German, French, Hindi, Arabic
  • Arabic → English, Persian
  • Persian → English

All pairs now produce complete output covering the full input. Before this
change every pair was truncated at roughly Section 03/04 of a 5-section test
document.

RTL languages (Arabic, Persian) are handled correctly, chunks are always
joined in forward document order since each chunk is translated independently.

Signed-off-by: RSKKSOFFICIAL <rsksofficial02@gmail.com>
Signed-off-by: RSKKSOFFICIAL <rsksofficial02@gmail.com>
Signed-off-by: RSKKSOFFICIAL <rsksofficial02@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support large input texts with more than 250 words

2 participants