This system does demonstrate°hard word the flaws°hard word of some of the smaller models. Gemma°hard word3:4b, when I asked it to define°hard word artillery°hard word, said°hard word that one definition°hard word was an alternative°hard word for artilegia°hard word, a (non-existent°hard word) word for "fingers or toes".
----
The next goal°hard word is to get a Chinese word-frequency°hard word distribution.
This isn't°hard word too hard.
- Download°hard word a zhwiki°hard word dump°hard word.
- Generate a list of the "top 2500" pages. 💡 I have some old code°hard word that does a PageRank-like°hard word algorithm°hard word to find the top pages. For enwiki°hard word, the results are quite good; with a few anomalies°hard word like Oxford°hard word University°hard word Press being highly ranked°hard word because of citations°hard word. ⚙️ it is convenient°hard word that most of the Wiki°hard word templates°hard word are in English°hard word.
- Use jieba°hard word to tokenize°hard word the text°hard word.
- Get a word count.
- "Merge°hard word" this with the English°hard word lists.
----
The obvious problem is that words don't°hard word match 1-1 between languages. eros°hard word, agape°hard word, and philia°hard word.
The details of these problems are unpredictable°hard word at this time.