{"chain":[{"channel":"cities","content":" <<< Edgeley, North Dakota, is a small rural town in LaMoure County, located in the southeastern part of the state. With a population hovering around 500 people, it's one of many prairie towns that exemplify the broader character of the upper Great Plains\u2014quiet, sparsely populated, and closely tied to agriculture. >>>\r\n\r\n----\r\n\r\nhttps://www.lesswrong.com/posts/bfHDoWLnBH9xR3YAK/ai-2027-is-a-bet-against-amdahl-s-law\r\n\r\n <<< Of course the post is right. The various << FOOM >> claims are all bullshit. And Amdahl's Law is one of the reason why. Just because a few things will be a hundred times faster (or a million times faster) doesn't make the whole thing that much faster. >>>\r\n\r\nAlso, AGI definitions vary so widely, from << things that have already happened >> to << things that are impossible >>, that a \"prediction market\" is nearly meaningless.\r\n\r\n----\r\n\r\nI have seen various commentary related to \"Twilight of the Edgelords\" ( https://www.astralcodexten.com/p/twilight-of-the-edgelords ), a piece that I don't have access to.\r\n\r\nAnd, the response I can piece together from the fragments I can see would fall under GUILD LAW. ( additional commentary at https://www.writingruxandrabio.com/p/the-edgelords-were-right-a-response and https://theahura.substack.com/p/contra-scott-and-rux-on-whos-to-blame )\r\n\r\n----\r\n\r\nhttps://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/\r\n\r\n<<< To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QAT) that dramatically reduces memory requirements while maintaining high quality. This enables you to run powerful models like Gemma 3 27B locally on consumer-grade GPUs like the NVIDIA RTX 3090. >>>\r\n\r\nIt seems pretty obvious. A majority of the users of open-source models are using quantized models on personal hardware; might as well optimize that use-case. ( it is less clear that a majority of the CPU cycles are there; but a majority of the people certainly are.)\r\n\r\nMy next round of updating the << Greenland metrics >> will have to include the gemma3-12b-qat model. ( or, maybe the 27b. According to Hacker News, gemma3-27b-Q4 << only uses ~22Gb (via Ollama) or ~15GB (MLX) >>. On a 24GB machine, this clearly needs the non-Ollama approach.)\r\n\r\nAnd, also, GPT-4.1 . And probably Gemini-2.5 . ( the goal for these models should be to perform at 100% accuracy.) ( well, actually, a few of the \"correct\" benchmark answers right now are incorrect.)","created_at":"2025-04-21T19:45:31.800653","id":353,"is_target":false,"parent_id":null,"processed_content":"

Edgeley, North Dakota, is a small rural town in LaMoure County, located in the southeastern part of the state. With a population hovering around 500 people, it's one of many prairie towns that exemplify the broader character of the upper Great Plains\u2014quiet, sparsely populated, and closely tied to agriculture.

\r

https://www.lesswrong.com/posts/bfHDoWLnBH9xR3YAK/ai-2027-is-a-bet-against-amdahl-s-law\r

\n

Of course the post is right. The various FOOM claims are all bullshit. And Amdahl's Law is one of the reason why. Just because a few things will be a hundred times faster (or a million times faster) doesn't make the whole thing that much faster.

\r

\n

Also, AGI definitions vary so widely, from things that have already happened to things that are impossible, that a \"prediction market\" is nearly meaningless.\r

I have seen various commentary related to \"Twilight of the Edgelords\" \n \u2699\ufe0f\n ( https://www.astralcodexten.com/p/twilight-of-the-edgelords )\n , a piece that I don't have access to.\r

\n

And, the response I can piece together from the fragments I can see would fall under GUILD LAW. \n \u2699\ufe0f\n ( additional commentary at https://www.writingruxandrabio.com/p/the-edgelords-were-right-a-response and https://theahura.substack.com/p/contra-scott-and-rux-on-whos-to-blame )\n \r

https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/\r

\n

To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QAT) that dramatically reduces memory requirements while maintaining high quality. This enables you to run powerful models like Gemma 3 27B locally on consumer-grade GPUs like the NVIDIA RTX 3090.

\r

\n

It seems pretty obvious. A majority of the users of open-source models are using quantized models on personal hardware; might as well optimize that use-case. \n \ud83d\udca1\n ( it is less clear that a majority of the CPU cycles are there; but a majority of the people certainly are.)\n \r

\n

My next round of updating the Greenland metrics will have to include the gemma3-12b-qat model. \n \ud83d\udca1\n ( or, maybe the 27b. According to Hacker News, gemma3-27b-Q4 only uses ~22Gb (via Ollama) or ~15GB (MLX). On a 24GB machine, this clearly needs the non-Ollama approach.)\n \r

\n

And, also, GPT-4.1 . And probably Gemini-2.5 . \n \ud83d\udca1\n ( the goal for these models should be to perform at 100% accuracy.)\n \n \u2694\ufe0f\n ( well, actually, a few of the \"correct\" benchmark answers right now are incorrect.)\n

","subject":"edgeley (part 1)"},{"channel":"cities","content":"so far today: running the \"proficiency\" benchmarks against GPT-4-1 and Gemini-2.5-flash.\r\n\r\n----\r\n\r\nThe headline: Google's cheap model can count letters. Gemini was substantially slower than both OpenAI and Anthropic (but, perhaps, that can vary day-to-day). But it got 96% on the infamous << count how many \"R\"s in strawberry >> metric, and none of the similarly-priced models got above 70%. ( the only metric it did \"bad\" on was the IPA one, and that is because the response normalization code is broken)\r\n\r\n----\r\n\r\nfor pricing ( all prices per million tokens):\r\n<<< GPT-4-1-nano: 10c IN, 40c OUT\r\nGPT-4-1-mini: 40c IN, 160c OUT\r\nGPT-4o-mini: 30c IN, 120c OUT\r\nGemini-2.5-flash: 15c IN, 60c OUT\r\nClaude-3-5-haiku: 80c IN, 400c OUT >>>\r\n\r\n Most of these have (or will have) \"cache\" discounts of 50-90% for repeated queries with the same long context.\r\n Claude is both the most expensive at this tier, and the lowest-performing. And the least-recently updated.\r\n presumably, they will have a new model at half the price, next week.","created_at":"2025-04-24T20:00:24.398343","id":356,"is_target":false,"parent_id":353,"processed_content":"

so far today: running the \"proficiency\" benchmarks against GPT-4-1 and Gemini-2.5-flash.\r

The headline: Google's cheap model can count letters. Gemini was substantially slower than both OpenAI and Anthropic (but, perhaps, that can vary day-to-day). But it got 96% on the infamous count how many \"R\"s in strawberry metric, and none of the similarly-priced models got above 70%. \n \ud83d\udca1\n ( the only metric it did \"bad\" on was the IPA one, and that is because the response normalization code is broken)\n \r

for pricing \n \u2699\ufe0f\n ( all prices per million tokens)\n :\r

\n

GPT-4-1-nano: 10c IN, 40c OUT\r

\n

GPT-4-1-mini: 40c IN, 160c OUT\r

\n

GPT-4o-mini: 30c IN, 120c OUT\r

\n

Gemini-2.5-flash: 15c IN, 60c OUT\r

\n

Claude-3-5-haiku: 80c IN, 400c OUT

\r

\n

\n \u2699\ufe0f\n Most of these have (or will have) \"cache\" discounts of 50-90% for repeated queries with the same long context.\r\n

\n

\n \ud83d\udca1\n Claude is both the most expensive at this tier, and the lowest-performing. And the least-recently updated.\r\n

\n

\n \ud83d\udd25\n presumably, they will have a new model at half the price, next week.\n

","subject":"edgeley (part 2)"},{"channel":"cities","content":"Last night, Claude wrote some code for the << qualification >> metrics. ( which might now be called *exemplar* tasks. The metric is \"respond to one prompt\".) ( to some degree, the goal is to test \"changes in context\" as much as \"changes in model\")\r\n\r\n----\r\n\r\nAn earlier task (from late 2023) was to answer the question: Who was << Pablo Arosemena >>? ( the Wikipedia article [[Pablo Arosemena]] is about an obscure politician from Panama)\r\n\r\nThe 8b models don't know who this is. But, they most commonly think he is an obscure painter. ( probably because of Pablo Picasso)\r\n\r\nIs there some sense this is a << true stereotype >>? Is it more likely he was a painter than that he was a baker, a masseuse, or a peasant farmer? ( well, actually ... it's more likely someone of this name had one of those jobs. But, less likely they were written about.)\r\n\r\n----\r\n\r\nI am getting a new computer. ( \"only\" $600) This should allow better speed comparisons between the models. ( the inconveniences of having an external USB drive, power demands, and heat creation \"on my lap\" grew to be too much.)\r\n\r\n----\r\n\r\nI need to do one more \"schema improvement\" pass on the \"dictionary\".\r\n\r\nThings like \"Chinese translation\", \"Korean translation\" need to be in a << dataclass >>, rather than passed as parameters everywhere.\r\n\r\nFor now, I want the indexes (and explicit NULL values), so these are database columns, rather than an \"all_translations\" JSON blob.\r\n\r\n----\r\n\r\nSome of the benchmarks should be re-written once the \"dictionary\" API is available.\r\n\r\n----\r\n\r\nClaude invented \"categories\" for the benchmarks: \"Language\", \"Reasoning\", \"Knowledge\", and \"Translation\". These are ... decent.\r\n\r\nBut the zeroth category is << token introspection >>. For \"how many letters is the word << triumphant >>\" questions. ( even the \"spell check\" tests that require repeating a misspelled word are probably in this category) ( you can have an LLM without << token introspection >>. but it should be very doable. possibly with some form of API / injection.)\r\n\r\nThen, *Language* (starting with definitions and antonyms), *Knowledge* (starting with basic geography), and *Translation* (starting with EN-FR, EN-ZH, SW-KO - word-based).\r\n\r\nThe only \"reasoning\" task so far might be \"unit conversion\". But that would have a different name.\r\n\r\n----\r\n\r\nThe \"translation\" tasks have to deal with the different vocabulary sizes of different languages.\r\n\r\nThis is one part << technical vocabulary >> ( does Swahili have a word for << capacitor >> that isn't a loan word?) and one part << eccentric distinctions >> ( Chinese has \u4e8c and \u4e24 for \"two\")\r\n\r\nSo far, I have largely mitigated this problem by avoiding it.\r\n\r\n----\r\n\r\nThe dictionary will need some type of \"class\" system. Specifically, I want to say \"get a random animal\" and have it do that.\r\n\r\nI am putting that off as well. Largely because it is a morass of taxonomical hell that has stymied decades of efforts.","created_at":"2025-04-28T17:04:56.472050","id":455,"is_target":true,"parent_id":356,"processed_content":"

Last night, Claude wrote some code for the qualification metrics. \n \ud83d\udca1\n ( which might now be called exemplar tasks. The metric is \"respond to one prompt\".)\n \n \ud83d\udd25\n ( to some degree, the goal is to test \"changes in context\" as much as \"changes in model\")\n \r

An earlier task (from late 2023) was to answer the question: Who was Pablo Arosemena? \n \u2699\ufe0f\n ( the Wikipedia article Pablo Arosemena is about an obscure politician from Panama)\n \r

\n

The 8b models don't know who this is. But, they most commonly think he is an obscure painter. \n \ud83d\udd25\n ( probably because of Pablo Picasso)\n \r

\n

Is there some sense this is a true stereotype? Is it more likely he was a painter than that he was a baker, a masseuse, or a peasant farmer? \n \u2694\ufe0f\n ( well, actually ... it's more likely someone of this name had one of those jobs. But, less likely they were written about.)\n \r

I am getting a new computer. \n \ud83d\udd25\n ( \"only\" $600)\n This should allow better speed comparisons between the models. \n \ud83d\udca1\n ( the inconveniences of having an external USB drive, power demands, and heat creation \"on my lap\" grew to be too much.)\n \r

I need to do one more \"schema improvement\" pass on the \"dictionary\".\r

\n

Things like \"Chinese translation\", \"Korean translation\" need to be in a dataclass, rather than passed as parameters everywhere.\r

\n

For now, I want the indexes (and explicit NULL values), so these are database columns, rather than an \"all_translations\" JSON blob.\r

Some of the benchmarks should be re-written once the \"dictionary\" API is available.\r

Claude invented \"categories\" for the benchmarks: \"Language\", \"Reasoning\", \"Knowledge\", and \"Translation\". These are ... decent.\r

\n

But the zeroth category is token introspection. For \"how many letters is the word triumphant\" questions. \n \ud83d\udca1\n ( even the \"spell check\" tests that require repeating a misspelled word are probably in this category)\n \n \u2699\ufe0f\n ( you can have an LLM without token introspection. but it should be very doable. possibly with some form of API / injection.)\n \r

\n

Then, Language (starting with definitions and antonyms), Knowledge (starting with basic geography), and Translation (starting with EN-FR, EN-ZH, SW-KO - word-based).\r

\n

The only \"reasoning\" task so far might be \"unit conversion\". But that would have a different name.\r

The \"translation\" tasks have to deal with the different vocabulary sizes of different languages.\r

\n

This is one part technical vocabulary \n \ud83d\udca1\n ( does Swahili have a word for capacitor that isn't a loan word?)\n and one part eccentric distinctions \n \u2699\ufe0f\n ( Chinese has \u4e8c and \u4e24 for \"two\")\n \r

\n

So far, I have largely mitigated this problem by avoiding it.\r

The dictionary will need some type of \"class\" system. Specifically, I want to say \"get a random animal\" and have it do that.\r

\n

I am putting that off as well. Largely because it is a morass of taxonomical hell that has stymied decades of efforts.

","subject":"edgeley (part 3)"}]}