There is a cmake option called MLC_ENABLE_SENTENCEPIECE_TOKENIZER but as today, when OFF, it does nt totally disable/skip sentencepiece, eg that line still links with the sentpiece lib:
|
target_link_libraries(tokenizers_cpp PRIVATE tokenizers_c sentencepiece-static ${TOKENIZERS_CPP_LINK_LIBS}) |
Would you mind if I add some small retouches to really fully disable sentpiece when that option is OFF?
Best regards
WT
There is a cmake option called MLC_ENABLE_SENTENCEPIECE_TOKENIZER but as today, when OFF, it does nt totally disable/skip sentencepiece, eg that line still links with the sentpiece lib:
tokenizers-cpp/CMakeLists.txt
Line 186 in f777109
Would you mind if I add some small retouches to really fully disable sentpiece when that option is OFF?
Best regards
WT