{"channel":"cities","content":"so far today: running the \"proficiency\" benchmarks against GPT-4-1 and Gemini-2.5-flash.\r\n\r\n----\r\n\r\nThe headline: Google's cheap model can count letters. Gemini was substantially slower than both OpenAI and Anthropic (but, perhaps, that can vary day-to-day). But it got 96% on the infamous << count how many \"R\"s in strawberry >> metric, and none of the similarly-priced models got above 70%. ( the only metric it did \"bad\" on was the IPA one, and that is because the response normalization code is broken)\r\n\r\n----\r\n\r\nfor pricing ( all prices per million tokens):\r\n<<< GPT-4-1-nano: 10c IN, 40c OUT\r\nGPT-4-1-mini: 40c IN, 160c OUT\r\nGPT-4o-mini: 30c IN, 120c OUT\r\nGemini-2.5-flash: 15c IN, 60c OUT\r\nClaude-3-5-haiku: 80c IN, 400c OUT >>>\r\n\r\n Most of these have (or will have) \"cache\" discounts of 50-90% for repeated queries with the same long context.\r\n Claude is both the most expensive at this tier, and the lowest-performing. And the least-recently updated.\r\n presumably, they will have a new model at half the price, next week.","created_at":"2025-04-24T20:00:24.398343","id":356,"llm_annotations":{},"parent_id":353,"processed_content":"

so far today: running the \"proficiency\" benchmarks against GPT-4-1 and Gemini-2.5-flash.\r

The headline: Google's cheap model can count letters. Gemini was substantially slower than both OpenAI and Anthropic (but, perhaps, that can vary day-to-day). But it got 96% on the infamous count how many \"R\"s in strawberry metric, and none of the similarly-priced models got above 70%. \n \ud83d\udca1\n ( the only metric it did \"bad\" on was the IPA one, and that is because the response normalization code is broken)\n \r

for pricing \n \u2699\ufe0f\n ( all prices per million tokens)\n :\r

\n

GPT-4-1-nano: 10c IN, 40c OUT\r

\n

GPT-4-1-mini: 40c IN, 160c OUT\r

\n

GPT-4o-mini: 30c IN, 120c OUT\r

\n

Gemini-2.5-flash: 15c IN, 60c OUT\r

\n

Claude-3-5-haiku: 80c IN, 400c OUT

\r

\n

\n \u2699\ufe0f\n Most of these have (or will have) \"cache\" discounts of 50-90% for repeated queries with the same long context.\r\n

\n

\n \ud83d\udca1\n Claude is both the most expensive at this tier, and the lowest-performing. And the least-recently updated.\r\n

\n

\n \ud83d\udd25\n presumably, they will have a new model at half the price, next week.\n

","quotes":[],"subject":"edgeley (part 2)"}