Last night, Claude°hard word wrote°hard word some code°hard word for the qualification°hard word metrics°hard word. 💡 which might now be called exemplar tasks. The metric°hard word is "respond°hard word to one prompt°hard word". 🔥 to some degree, the goal°hard word is to test "changes in context°hard word" as°hard word much as°hard word "changes in model"
----
An earlier task (from late 2023) was to answer the question: Who was Pablo°hard word Arosemena°hard word? ⚙️ the Wikipedia°hard word article°hard word Pablo°hard word Arosemena°hard word is about an obscure°hard word politician°hard word from Panama°hard word
The 8b models don't°hard word know who this is. But, they most commonly think he is an obscure°hard word painter. 🔥 probably because of Pablo°hard word Picasso°hard word
Is there some sense this is a true stereotype°hard word? Is it more likely he was a painter than that he was a baker, a masseuse°hard word, or a peasant°hard word farmer? ⚔️ well, actually ... it's°hard word more likely someone of this name had one of those jobs. But, less likely they were written°hard word about.
----
I am getting a new computer. 🔥 "only" $600 This should allow better speed comparisons between the models. 💡 the inconveniences°hard word of having an external°hard word USB°hard word drive, power demands°hard word, and heat creation°hard word "on my lap" grew°hard word to be too much.
----
I need to do one more "schema°hard word improvement°hard word" pass on the "dictionary°hard word".
Things like "Chinese°hard word translation°hard word", "Korean°hard word translation°hard word" need to be in a dataclass°hard word, rather than passed as°hard word parameters°hard word everywhere.
For now, I want the indexes°hard word (and explicit°hard word NULL°hard word values), so these are database columns°hard word, rather than an "all_translations°hard word" JSON°hard word blob°hard word.
----
Some of the benchmarks°hard word should be re-written°hard word once the "dictionary°hard word" API°hard word is available°hard word.
----
Claude°hard word invented "categories°hard word" for the benchmarks°hard word: "Language", "Reasoning", "Knowledge", and "Translation°hard word". These are ... decent°hard word.
But the zeroth°hard word category°hard word is token°hard word introspection°hard word. For "how many letters is the word triumphant°hard word" questions. 💡 even the "spell check" tests that require°hard word repeating°hard word a misspelled°hard word word are probably in this category°hard word ⚙️ you can have an LLM°hard word without token°hard word introspection°hard word. but it should be very doable°hard word. possibly with some form of API°hard word / injection°hard word.
Then°hard word, Language (starting with definitions°hard word and antonyms°hard word), Knowledge (starting with basic°hard word geography), and Translation (starting with EN-FR°hard word, EN-ZH°hard word, SW-KO°hard word - word-based°hard word).
The only "reasoning" task so far might be "unit conversion°hard word". But that would have a different name.
----
The "translation°hard word" tasks have to deal with the different vocabulary°hard word sizes of different languages.
This is one part technical°hard word vocabulary°hard word 💡 does Swahili°hard word have a word for capacitor°hard word that isn't°hard word a loan°hard word word? and one part eccentric°hard word distinctions°hard word ⚙️ Chinese°hard word has 二 and 两 for "two"
So far, I have largely mitigated°hard word this problem by avoiding°hard word it.
----
The dictionary°hard word will need some type of "class" system. Specifically°hard word, I want to say "get a random animal" and have it do that.
I am putting°hard word that off as°hard word well. Largely because it is a morass°hard word of taxonomical°hard word hell°hard word that has stymied°hard word decades of efforts.