{"chain":[{"channel":"cities","content":" Jamestown, North Dakota, is located along I-94 in the eastern half of the state.\r\n\r\nToday's focus is on \"word frequency\".\r\n\r\n----\r\n\r\n <<< I started with two corpuses: one of 19th century literature (from Project Gutenberg), one of 20th century \"sci-fi\" literature. I got a rough word-rank for each, and combined them ( using the harmonic mean) to get a combined word-list. >>>\r\n\r\n <<< While many high-frequency function words such as *the*, *and*, and *of* maintain consistent rankings, others like *said*, *her*, *she*, and *me* show substantial divergence, suggesting notable stylistic or thematic shifts between the two periods and genres. >>>\r\n\r\n----\r\n\r\nIt is also a word-list at all. Some of the notes:\r\n\r\n# The word \"whale\" shows up a lot more in the 19th century corpus. This is because one of the books is [[Moby Dick]].\r\n# I am hoping to run an exhaustive listing of a few attributes. These include:\r\n> polysemy. ( I am less concerned with words like << get >> which have so many meanings as-to be indefinable, but instead words like << saw >> (\u770b or \u952f\u5b50) or << face >> (\u9762\u5411 or \u8138))\r\n> by lemma. \"went\" (108th) v. \"go\" (80th).\r\n> by part-of-speech. defined as \"what the LLMs define as part-of-speech\".\r\n> a \"second-level\" of word-type details.","created_at":"2025-04-06T20:39:52.199751","id":343,"is_target":false,"parent_id":null,"processed_content":"

\n \u2699\ufe0f\n Jamestown, North Dakota, is located along I-94 in the eastern half of the state.\r\n

\n

Today's focus is on \"word frequency\".\r

I started with two corpuses: one of 19th century literature (from Project Gutenberg), one of 20th century \"sci-fi\" literature. I got a rough word-rank for each, and combined them \n \u2699\ufe0f\n ( using the harmonic mean)\n to get a combined word-list.

\r

\n

While many high-frequency function words such as the, and, and of maintain consistent rankings, others like said, her, she, and me show substantial divergence, suggesting notable stylistic or thematic shifts between the two periods and genres.

\r

It is also a word-list at all. Some of the notes:\r

\n
    \n
  • The word \"whale\" shows up a lot more in the 19th century corpus. This is because one of the books is Moby Dick.\r
    • \n
  • I am hoping to run an exhaustive listing of a few attributes. These include:\r
    • \n
\n
    \n
  • polysemy. \n \u2699\ufe0f\n ( I am less concerned with words like get which have so many meanings as-to be indefinable, but instead words like saw (\u770b or \u952f\u5b50) or face (\u9762\u5411 or \u8138))\n \r
    • \n
  • by lemma. \"went\" (108th) v. \"go\" (80th).\r
    • \n
  • by part-of-speech. defined as \"what the LLMs define as part-of-speech\".\r
    • \n
  • a \"second-level\" of word-type details.
    • \n
","subject":"jamestown (part 1)"},{"channel":"cities","content":"This seems to be \"evolving\" into a dictionary. Somewhat by accident; the *machine* wrote a command-line frontend for me (without my asking) that looks a *lot* like a dictionary.\r\n\r\n----\r\n\r\nWhen are two definitions the same? ( when they are not *different*. and they are different if the part-of-speech differs, or the Chinese translation differs. and some other times.)\r\n\r\n----\r\n\r\nThese three seem to be the same.\r\n\r\n<<< Word: eternal\r\nRank: 2547\r\n\r\nDefinitions:\r\n [1] Existing or continuing without end; everlasting.\r\n Confidence: 0.95\r\n Part of speech: adjective\r\n Lemma: eternal\r\n Chinese: \u6c38\u6052\r\n Subtype: duration\r\n Notes: Often used in philosophical or religious contexts to describe the nature of existence or the divine.\r\n Examples:\r\n - The concept of eternal life is central to many religions.\r\n\r\n [2] Lasting or existing forever; without beginning or end.\r\n Confidence: 0.90\r\n Part of speech: adjective\r\n Lemma: eternal\r\n Chinese: \u6c38\u6052\r\n Subtype: duration\r\n Notes: Can refer to abstract ideas, such as love or truth, that are considered timeless.\r\n Examples:\r\n - Her love for him felt eternal, transcending time and space.\r\n\r\n [3] Relating to or being a part of eternity; timeless.\r\n Confidence: 0.85\r\n Part of speech: adjective\r\n Lemma: eternal\r\n Chinese: \u6c38\u6052\r\n Subtype: duration\r\n Notes: Often used in literary or poetic contexts to evoke a sense of infinity.\r\n Examples:\r\n - The stars in the night sky seemed to whisper eternal secrets.\r\n>>>\r\n\r\nOn the other hand, these three might be different:\r\n\r\n<<< Word: transform\r\nRank: 2646\r\n\r\nDefinitions:\r\n [1] To change in form, appearance, or structure; to metamorphose.\r\n Confidence: 0.95\r\n Part of speech: verb\r\n Lemma: transform\r\n Chinese: \u53d8\u5f62\r\n Subtype: change\r\n Notes: Often used in contexts involving significant change or conversion.\r\n Examples:\r\n - The caterpillar will transform into a butterfly.\r\n\r\n [2] To cause to change in character or condition; to convert.\r\n Confidence: 0.90\r\n Part of speech: verb\r\n Lemma: transform\r\n Chinese: \u8f6c\u53d8\r\n Subtype: change\r\n Notes: Commonly used in contexts such as technology, business, or personal development.\r\n Examples:\r\n - The new software will transform the way we manage our projects.\r\n\r\n [3] In mathematics, to change the coordinates of a point or the representation of a function.\r\n Confidence: 0.85\r\n Part of speech: verb\r\n Lemma: transform\r\n Chinese: \u53d8\u6362\r\n Subtype: change\r\n Notes: Used in various branches of mathematics, including geometry and calculus.\r\n Examples:\r\n - We can transform the equation into a simpler form. >>>\r\n\r\n----\r\n\r\nand these are *definitely* different:\r\n\r\n<<< [1] A piece of furniture with a flat top and one or more legs, used for placing items on or for working at.\r\n Confidence: 1.00\r\n Part of speech: noun\r\n Lemma: table\r\n Chinese: \u684c\u5b50\r\n Subtype: small_movable_object\r\n Notes: Commonly used in homes and offices.\r\n Examples:\r\n - We gathered around the dining table for dinner.\r\n\r\n [2] A systematic arrangement of data, usually in rows and columns, for easy reference.\r\n Confidence: 1.00\r\n Part of speech: noun\r\n Lemma: table\r\n Chinese: \u8868\u683c\r\n Subtype: small_movable_object\r\n Notes: Often used in academic and professional contexts.\r\n Examples:\r\n - The research paper included a table summarizing the results of the experiment.\r\n\r\n [3] To postpone consideration of a motion or proposal in a meeting or legislative assembly.\r\n Confidence: 0.90\r\n Part of speech: verb\r\n Lemma: table\r\n Chinese: \u6401\u7f6e\r\n Subtype: other\r\n Notes: Usage can vary by region; in some places, it means to bring a motion forward for discussion.\r\n Examples:\r\n - The committee decided to table the discussion until next week.\r\n>>>","created_at":"2025-04-07T18:21:15.867594","id":344,"is_target":false,"parent_id":343,"processed_content":"

This seems to be \"evolving\" into a dictionary. Somewhat by accident; the machine wrote a command-line frontend for me (without my asking) that looks a lot like a dictionary.\r

When are two definitions the same? \n \ud83d\udd25\n ( when they are not different. and they are different if the part-of-speech differs, or the Chinese translation differs. and some other times.)\n \r

These three seem to be the same.\r

\n

Word: eternal\r

\n

Rank: 2547\r

\n

\r

\n

Definitions:\r

\n

[1] Existing or continuing without end; everlasting.\r

\n

Confidence: 0.95\r

\n

Part of speech: adjective\r

\n

Lemma: eternal\r

\n

Chinese: \u6c38\u6052\r

\n

Subtype: duration\r

\n

Notes: Often used in philosophical or religious contexts to describe the nature of existence or the divine.\r

\n

Examples:\r

\n

- The concept of eternal life is central to many religions.\r

\n

\r

\n

[2] Lasting or existing forever; without beginning or end.\r

\n

Confidence: 0.90\r

\n

Part of speech: adjective\r

\n

Lemma: eternal\r

\n

Chinese: \u6c38\u6052\r

\n

Subtype: duration\r

\n

Notes: Can refer to abstract ideas, such as love or truth, that are considered timeless.\r

\n

Examples:\r

\n

- Her love for him felt eternal, transcending time and space.\r

\n

\r

\n

[3] Relating to or being a part of eternity; timeless.\r

\n

Confidence: 0.85\r

\n

Part of speech: adjective\r

\n

Lemma: eternal\r

\n

Chinese: \u6c38\u6052\r

\n

Subtype: duration\r

\n

Notes: Often used in literary or poetic contexts to evoke a sense of infinity.\r

\n

Examples:\r

\n

- The stars in the night sky seemed to whisper eternal secrets.\r

\r

\n

On the other hand, these three might be different:\r

\n

Word: transform\r

\n

Rank: 2646\r

\n

\r

\n

Definitions:\r

\n

[1] To change in form, appearance, or structure; to metamorphose.\r

\n

Confidence: 0.95\r

\n

Part of speech: verb\r

\n

Lemma: transform\r

\n

Chinese: \u53d8\u5f62\r

\n

Subtype: change\r

\n

Notes: Often used in contexts involving significant change or conversion.\r

\n

Examples:\r

\n

- The caterpillar will transform into a butterfly.\r

\n

\r

\n

[2] To cause to change in character or condition; to convert.\r

\n

Confidence: 0.90\r

\n

Part of speech: verb\r

\n

Lemma: transform\r

\n

Chinese: \u8f6c\u53d8\r

\n

Subtype: change\r

\n

Notes: Commonly used in contexts such as technology, business, or personal development.\r

\n

Examples:\r

\n

- The new software will transform the way we manage our projects.\r

\n

\r

\n

[3] In mathematics, to change the coordinates of a point or the representation of a function.\r

\n

Confidence: 0.85\r

\n

Part of speech: verb\r

\n

Lemma: transform\r

\n

Chinese: \u53d8\u6362\r

\n

Subtype: change\r

\n

Notes: Used in various branches of mathematics, including geometry and calculus.\r

\n

Examples:\r

\n

- We can transform the equation into a simpler form.

\r

and these are definitely different:\r

\n

[1] A piece of furniture with a flat top and one or more legs, used for placing items on or for working at.\r

\n

Confidence: 1.00\r

\n

Part of speech: noun\r

\n

Lemma: table\r

\n

Chinese: \u684c\u5b50\r

\n

Subtype: small_movable_object\r

\n

Notes: Commonly used in homes and offices.\r

\n

Examples:\r

\n

- We gathered around the dining table for dinner.\r

\n

\r

\n

[2] A systematic arrangement of data, usually in rows and columns, for easy reference.\r

\n

Confidence: 1.00\r

\n

Part of speech: noun\r

\n

Lemma: table\r

\n

Chinese: \u8868\u683c\r

\n

Subtype: small_movable_object\r

\n

Notes: Often used in academic and professional contexts.\r

\n

Examples:\r

\n

- The research paper included a table summarizing the results of the experiment.\r

\n

\r

\n

[3] To postpone consideration of a motion or proposal in a meeting or legislative assembly.\r

\n

Confidence: 0.90\r

\n

Part of speech: verb\r

\n

Lemma: table\r

\n

Chinese: \u6401\u7f6e\r

\n

Subtype: other\r

\n

Notes: Usage can vary by region; in some places, it means to bring a motion forward for discussion.\r

\n

Examples:\r

\n

- The committee decided to table the discussion until next week.\r

","subject":"jamestown (part 2)"},{"channel":"cities","content":"the project has been at the << liminal >> point between \"interesting\" and \"we already have dictionaries\".\r\n\r\nsome things are easier for the LLM than others. for \"get an IPA pronunciation\", I eventually determined that the best choice was to give up and download a flat-text file. (and, maybe, get the *machine* to deal with the ambiguous cases)\r\n\r\n----\r\n\r\nthe \"merge various word-frequency lists into one list\" code, after a few rounds of telling the *machine* it was wrong, now works fine.\r\n\r\n the top-level differences are predictable. words like << you >> are less common on Wikipedia.\r\n because of some bug, words like \"vernacular\" and \"justification\" are showing up in the top 250. also, not all the lists manage contractions correctly, giving \"words\" like << isn >> and << doesn >> ('t).\r\n\r\n----\r\n\r\nperhaps the next task is \"generate an annotated version of text, where the text is *colored* based on the word frequency\".\r\n and, possibly, the uncommon words get Chinese translations added.\r\n that will certainly help the English monoglot who is confused!\r\n\r\n----\r\n\r\nthe variance in the word frequencies (mostly) says something about the \"cultural loading\" of the words.\r\n\r\nmost of the words that are more common in 19th century books are low-cultural-loading. ( perhaps the prevalence of << gentleman >> is cultural. but words like << rain >> and << hat >> are more frequent just because other words (<< geometry >>, << organic >>) are less frequent.)","created_at":"2025-04-12T15:42:24.374769","id":347,"is_target":false,"parent_id":344,"processed_content":"

the project has been at the liminal point between \"interesting\" and \"we already have dictionaries\".\r

\n

some things are easier for the LLM than others. for \"get an IPA pronunciation\", I eventually determined that the best choice was to give up and download a flat-text file. (and, maybe, get the machine to deal with the ambiguous cases)\r

the \"merge various word-frequency lists into one list\" code, after a few rounds of telling the machine it was wrong, now works fine.\r

\n

\n \ud83d\udca1\n the top-level differences are predictable. words like you are less common on Wikipedia.\r\n

\n

\n \u2699\ufe0f\n because of some bug, words like \"vernacular\" and \"justification\" are showing up in the top 250. also, not all the lists manage contractions correctly, giving \"words\" like isn and doesn ('t).\r\n

perhaps the next task is \"generate an annotated version of text, where the text is colored based on the word frequency\".\r

\n

\n \ud83d\udca1\n and, possibly, the uncommon words get Chinese translations added.\r\n

\n

\n \ud83d\udd25\n that will certainly help the English monoglot who is confused!\r\n

the variance in the word frequencies (mostly) says something about the \"cultural loading\" of the words.\r

\n

most of the words that are more common in 19th century books are low-cultural-loading. \n \ud83d\udca1\n ( perhaps the prevalence of gentleman is cultural. but words like rain and hat are more frequent just because other words (geometry, organic) are less frequent.)\n

","subject":"jamestown (part 3)"},{"channel":"cities","content":"This system does demonstrate the flaws of some of the smaller models. Gemma3:4b, when I asked it to define << artillery >>, said that one definition was an alternative for << artilegia >>, a (non-existent) word for \"fingers or toes\".\r\n\r\n----\r\n\r\nThe next goal is to get a *Chinese* word-frequency distribution.\r\n\r\nThis isn't too hard.\r\n\r\n# Download a zhwiki dump.\r\n# Generate a list of the \"top 2500\" pages. ( I have some old code that does a PageRank-like algorithm to find the top pages. For << enwiki >>, the results are quite good; with a few anomalies like [[Oxford University Press]] being highly ranked because of citations.) ( it is convenient that most of the Wiki templates are in English.)\r\n# Use << jieba >> to tokenize the text.\r\n# Get a word count.\r\n# \"Merge\" this with the English lists.\r\n\r\n----\r\n\r\nThe obvious problem is that words don't match 1-1 between languages. << eros >>, << agape >>, and << philia >>.\r\n\r\nThe details of these problems are unpredictable at this time.\r\n\r\n----","created_at":"2025-04-13T18:27:23.864613","id":349,"is_target":false,"parent_id":347,"processed_content":"

This system does demonstrate the flaws of some of the smaller models. Gemma3:4b, when I asked it to define artillery, said that one definition was an alternative for artilegia, a (non-existent) word for \"fingers or toes\".\r

The next goal is to get a Chinese word-frequency distribution.\r

\n

This isn't too hard.\r

\n
    \n
  • Download a zhwiki dump.\r
    • \n
  • Generate a list of the \"top 2500\" pages. \n \ud83d\udca1\n ( I have some old code that does a PageRank-like algorithm to find the top pages. For enwiki, the results are quite good; with a few anomalies like Oxford University Press being highly ranked because of citations.)\n \n \u2699\ufe0f\n ( it is convenient that most of the Wiki templates are in English.)\n \r
    • \n
  • Use jieba to tokenize the text.\r
    • \n
  • Get a word count.\r
    • \n
  • \"Merge\" this with the English lists.\r
    • \n

The obvious problem is that words don't match 1-1 between languages. eros, agape, and philia.\r

\n

The details of these problems are unpredictable at this time.\r

","subject":"jamestown (part 4)"},{"channel":"cities","content":"5/5\r\n\r\nit has been a drudge to get work done.\r\n\r\n----\r\n\r\nthe end-goal is within sight. one more round of prompt-tuning, a $1 gpt-4.1-nano run, and some rounds of \"LLM consensus checking\".\r\n\r\nthe goal then becomes *applications*.\r\n\r\n# LLM benchmarks. ( use the *machine* to feed the *machine*) ( \"which word has this definition\" questions will be possible at some point. but not yet.)\r\n# Elementary education. ( which words should a 3rd/5th grader know? be studying?) ( so far no useful progress on \"how easy/hard is it to spell this word\")\r\n# Second-language learning. ( a \"which is the Chinese for this word in this sentence\" app.)\r\n# Text difficulty. A smarter metric than Flesch-Kincaid.\r\n\r\n----\r\n\r\nThe \"cosine similarity\" question of \"how similar are these word definitions\" is not yet solved. I'm not sure I can solve it.\r\n\r\nI can test it; I have several ways of generating embeddings. And (probably) these can include a sentence as context.\r\n\r\n----\r\n\r\nI also have no solutions for the \"group different word-forms with the same meaning\". For << jump >>, << jumps >>, << jumped >>, for example.\r\n\r\nThis would be a much more substantial problem for more highly-conjugated languages. With English, it is almost avoidable.\r\n\r\n----\r\n\r\nAround word 5000, i am seeing << raft >>, << yield >>, << algebra >>, and << pizza >>.\r\n\r\nThis seems correct enough? \"Algebra\" is more common in encyclopedic contexts, and \"pizza\" doesn't show up in the 19th century corpus at all.\r\n\r\n----\r\n\r\nBut, as far as \"exploration\" is concerned, I am reaching diminishing returns.\r\n\r\nI have one more list of \"see if Claude can do this quickly\". After that, \"glenora\" will become an inactive project.","created_at":"2025-04-16T18:57:28.551568","id":351,"is_target":true,"parent_id":349,"processed_content":"

5/5\r

\n

it has been a drudge to get work done.\r

the end-goal is within sight. one more round of prompt-tuning, a $1 gpt-4.1-nano run, and some rounds of \"LLM consensus checking\".\r

\n

the goal then becomes applications.\r

\n
    \n
  • LLM benchmarks. \n \ud83d\udd25\n ( use the machine to feed the machine)\n \n \ud83d\udca1\n ( \"which word has this definition\" questions will be possible at some point. but not yet.)\n \r
    • \n
  • Elementary education. \n \ud83d\udca1\n ( which words should a 3rd/5th grader know? be studying?)\n \n \u2699\ufe0f\n ( so far no useful progress on \"how easy/hard is it to spell this word\")\n \r
    • \n
  • Second-language learning. \n \ud83d\udca1\n ( a \"which is the Chinese for this word in this sentence\" app.)\n \r
    • \n
  • Text difficulty. A smarter metric than Flesch-Kincaid.\r
    • \n

The \"cosine similarity\" question of \"how similar are these word definitions\" is not yet solved. I'm not sure I can solve it.\r

\n

I can test it; I have several ways of generating embeddings. And (probably) these can include a sentence as context.\r

I also have no solutions for the \"group different word-forms with the same meaning\". For jump, jumps, jumped, for example.\r

\n

This would be a much more substantial problem for more highly-conjugated languages. With English, it is almost avoidable.\r

Around word 5000, i am seeing raft, yield, algebra, and pizza.\r

\n

This seems correct enough? \"Algebra\" is more common in encyclopedic contexts, and \"pizza\" doesn't show up in the 19th century corpus at all.\r

But, as far as \"exploration\" is concerned, I am reaching diminishing returns.\r

\n

I have one more list of \"see if Claude can do this quickly\". After that, \"glenora\" will become an inactive project.

","subject":"jamestown (part 5)"}]}