{"channel":"cities","content":"Last night, Claude wrote some code for the << qualification >> metrics. ( which might now be called *exemplar* tasks. The metric is \"respond to one prompt\".) ( to some degree, the goal is to test \"changes in context\" as much as \"changes in model\")\r\n\r\n----\r\n\r\nAn earlier task (from late 2023) was to answer the question: Who was << Pablo Arosemena >>? ( the Wikipedia article [[Pablo Arosemena]] is about an obscure politician from Panama)\r\n\r\nThe 8b models don't know who this is. But, they most commonly think he is an obscure painter. ( probably because of Pablo Picasso)\r\n\r\nIs there some sense this is a << true stereotype >>? Is it more likely he was a painter than that he was a baker, a masseuse, or a peasant farmer? ( well, actually ... it's more likely someone of this name had one of those jobs. But, less likely they were written about.)\r\n\r\n----\r\n\r\nI am getting a new computer. ( \"only\" $600) This should allow better speed comparisons between the models. ( the inconveniences of having an external USB drive, power demands, and heat creation \"on my lap\" grew to be too much.)\r\n\r\n----\r\n\r\nI need to do one more \"schema improvement\" pass on the \"dictionary\".\r\n\r\nThings like \"Chinese translation\", \"Korean translation\" need to be in a << dataclass >>, rather than passed as parameters everywhere.\r\n\r\nFor now, I want the indexes (and explicit NULL values), so these are database columns, rather than an \"all_translations\" JSON blob.\r\n\r\n----\r\n\r\nSome of the benchmarks should be re-written once the \"dictionary\" API is available.\r\n\r\n----\r\n\r\nClaude invented \"categories\" for the benchmarks: \"Language\", \"Reasoning\", \"Knowledge\", and \"Translation\". These are ... decent.\r\n\r\nBut the zeroth category is << token introspection >>. For \"how many letters is the word << triumphant >>\" questions. ( even the \"spell check\" tests that require repeating a misspelled word are probably in this category) ( you can have an LLM without << token introspection >>. but it should be very doable. possibly with some form of API / injection.)\r\n\r\nThen, *Language* (starting with definitions and antonyms), *Knowledge* (starting with basic geography), and *Translation* (starting with EN-FR, EN-ZH, SW-KO - word-based).\r\n\r\nThe only \"reasoning\" task so far might be \"unit conversion\". But that would have a different name.\r\n\r\n----\r\n\r\nThe \"translation\" tasks have to deal with the different vocabulary sizes of different languages.\r\n\r\nThis is one part << technical vocabulary >> ( does Swahili have a word for << capacitor >> that isn't a loan word?) and one part << eccentric distinctions >> ( Chinese has \u4e8c and \u4e24 for \"two\")\r\n\r\nSo far, I have largely mitigated this problem by avoiding it.\r\n\r\n----\r\n\r\nThe dictionary will need some type of \"class\" system. Specifically, I want to say \"get a random animal\" and have it do that.\r\n\r\nI am putting that off as well. Largely because it is a morass of taxonomical hell that has stymied decades of efforts.","created_at":"2025-04-28T17:04:56.472050","id":455,"llm_annotations":{},"parent_id":356,"processed_content":"

Last night, Claude wrote some code for the qualification metrics. \n \ud83d\udca1\n ( which might now be called exemplar tasks. The metric is \"respond to one prompt\".)\n \n \ud83d\udd25\n ( to some degree, the goal is to test \"changes in context\" as much as \"changes in model\")\n \r

An earlier task (from late 2023) was to answer the question: Who was Pablo Arosemena? \n \u2699\ufe0f\n ( the Wikipedia article Pablo Arosemena is about an obscure politician from Panama)\n \r

\n

The 8b models don't know who this is. But, they most commonly think he is an obscure painter. \n \ud83d\udd25\n ( probably because of Pablo Picasso)\n \r

\n

Is there some sense this is a true stereotype? Is it more likely he was a painter than that he was a baker, a masseuse, or a peasant farmer? \n \u2694\ufe0f\n ( well, actually ... it's more likely someone of this name had one of those jobs. But, less likely they were written about.)\n \r

I am getting a new computer. \n \ud83d\udd25\n ( \"only\" $600)\n This should allow better speed comparisons between the models. \n \ud83d\udca1\n ( the inconveniences of having an external USB drive, power demands, and heat creation \"on my lap\" grew to be too much.)\n \r

I need to do one more \"schema improvement\" pass on the \"dictionary\".\r

\n

Things like \"Chinese translation\", \"Korean translation\" need to be in a dataclass, rather than passed as parameters everywhere.\r

\n

For now, I want the indexes (and explicit NULL values), so these are database columns, rather than an \"all_translations\" JSON blob.\r

Some of the benchmarks should be re-written once the \"dictionary\" API is available.\r

Claude invented \"categories\" for the benchmarks: \"Language\", \"Reasoning\", \"Knowledge\", and \"Translation\". These are ... decent.\r

\n

But the zeroth category is token introspection. For \"how many letters is the word triumphant\" questions. \n \ud83d\udca1\n ( even the \"spell check\" tests that require repeating a misspelled word are probably in this category)\n \n \u2699\ufe0f\n ( you can have an LLM without token introspection. but it should be very doable. possibly with some form of API / injection.)\n \r

\n

Then, Language (starting with definitions and antonyms), Knowledge (starting with basic geography), and Translation (starting with EN-FR, EN-ZH, SW-KO - word-based).\r

\n

The only \"reasoning\" task so far might be \"unit conversion\". But that would have a different name.\r

The \"translation\" tasks have to deal with the different vocabulary sizes of different languages.\r

\n

This is one part technical vocabulary \n \ud83d\udca1\n ( does Swahili have a word for capacitor that isn't a loan word?)\n and one part eccentric distinctions \n \u2699\ufe0f\n ( Chinese has \u4e8c and \u4e24 for \"two\")\n \r

\n

So far, I have largely mitigated this problem by avoiding it.\r

The dictionary will need some type of \"class\" system. Specifically, I want to say \"get a random animal\" and have it do that.\r

\n

I am putting that off as well. Largely because it is a morass of taxonomical hell that has stymied decades of efforts.

","quotes":[],"subject":"edgeley (part 3)"}