Adding support for unigram sentencepiece model #51
Adding support for unigram sentencepiece model #51Ayushk4 merged 36 commits intoJuliaText:masterfrom
Conversation
|
|
Hi @Ayushk4, |
|
Travis CI fails on Julia 1.0, due to new DataDeps as dependency - requires updating Project.toml. |
|
@oxinabox Should we drop support for 0.7 tests now in Appveyor? |
|
Since DataDeps have been newly added, Travis and Appveyor also need to be updated to allow DataDeps downloads |
Yes |
Ayushk4
left a comment
There was a problem hiding this comment.
Please look into setting DATADEPS variable for Travis and Appveyor CI. Also remove 0.7 Julia tests for appveyor.
Co-authored-by: Ayush Kaushal <ayushk4@gmail.com>
Co-authored-by: Ayush Kaushal <ayushk4@gmail.com>
Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>
Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>
Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>
|
Thank you. |
As discussed in #44
I am adding support for the sentencepiece unigram model with
DataDepsto download pre-trained vocab file with their log-likelihoodfor more detail refer to the blog post here
Roadmap