{"channel":"cities","content":"This system does demonstrate the flaws of some of the smaller models. Gemma3:4b, when I asked it to define << artillery >>, said that one definition was an alternative for << artilegia >>, a (non-existent) word for \"fingers or toes\".\r\n\r\n----\r\n\r\nThe next goal is to get a *Chinese* word-frequency distribution.\r\n\r\nThis isn't too hard.\r\n\r\n# Download a zhwiki dump.\r\n# Generate a list of the \"top 2500\" pages. ( I have some old code that does a PageRank-like algorithm to find the top pages. For << enwiki >>, the results are quite good; with a few anomalies like [[Oxford University Press]] being highly ranked because of citations.) ( it is convenient that most of the Wiki templates are in English.)\r\n# Use << jieba >> to tokenize the text.\r\n# Get a word count.\r\n# \"Merge\" this with the English lists.\r\n\r\n----\r\n\r\nThe obvious problem is that words don't match 1-1 between languages. << eros >>, << agape >>, and << philia >>.\r\n\r\nThe details of these problems are unpredictable at this time.\r\n\r\n----","created_at":"2025-04-13T18:27:23.864613","id":349,"llm_annotations":{},"parent_id":347,"processed_content":"

This system does demonstrate the flaws of some of the smaller models. Gemma3:4b, when I asked it to define artillery, said that one definition was an alternative for artilegia, a (non-existent) word for \"fingers or toes\".\r

The next goal is to get a Chinese word-frequency distribution.\r

\n

This isn't too hard.\r

\n

The obvious problem is that words don't match 1-1 between languages. eros, agape, and philia.\r

\n

The details of these problems are unpredictable at this time.\r

","quotes":[],"subject":"jamestown (part 4)"}