Table of Contents One Eye, Zero Excuses What I Benchmarked Why Georgian? What I Built The Plan vs Reality The Five Tasks How It Runs on Kaggle Obstacles and Workarounds Models Tested Findings Level by Level Finding 1: Understanding Georgian Is Solved. Producing It Isn't. Finding 2: Numbers Are the Final Boss Finding 3: The Mistakes Are Very Georgian Finding 4: Georgian Costs More Tokens for Claude Finding 5: Cheap Can Be Smart Side Quest: The Uninvited Guest What It All Cost The Dashboard Limitations What's Next My Benchmark Wrapping Up
I really wanted to record a video for this one, just like I always do for my challenges. But yesterday I had a small surgery on my eyelid, and right now one of my eyes is swollen and patched up. Here's the current state:
Yep. I look like a pirate now. All I'm missing is a parrot and a cool nickname (suggestions are welcome in the comments).
But don't worry (if you were going to): I'm totally fine and my eyesight is still eagle-like from both eyes. It was just a small bug fix in the eyelid module. The eye itself is still in production and running smoothly.
That said, finishing a project with one-eye-patched-up is way harder than it sounds. Depth perception? Gone. Reading code? Awful. I don't believe in curses, but every time I join a challenge, something happens. If someone out there cursed me, please, just lift it. Give me a break! I just wanna work goddammit!
By the way shout out to my brother for chauffeuring me home from the hospital, because one-eye-patched-up driving felt like a really bad idea.
But a deadline is a deadline, and professionals deliver, no matter what. So I gave myself a little pep talk: "Dude, Michael Bisping became a UFC champion with basically one working eye. You can finish a coding project. No excuses, you're not lame." And so I did.
I benchmarked OpenAI's and Anthropic's models on how well they understand and produce Georgian. The whole process turned into a real adventure, but before we get to that, let me tell you why I picked this topic.
Georgian is one of the oldest languages in the world. It has its own alphabet and doesn't belong to any of the major language families. It's totally unique. Phonetically, it sometimes kind of sounds like Italian, because both languages are obsessed with vowels, but that's where the similarity ends. And vowel obsession doesn't mean weak consonants. Quite the contrary: Georgian has consonants most people have never heard, stacked in clusters that make everyone struggle (try saying გვფრცქვნი, with eight consonants in a row; it basically means "you peel us", and yes, I know it sounds like nonsense, but technically that word exists). It's an incredibly rich language.
Grammatically, it's ridiculously tough. I take it for granted, because first, I'm Georgian, and second, I'm a language nerd (even when I'm learning other languages). I know Georgian grammar well, but even most Georgians don't know every detail of the Georgian grammar, and I know almost nobody who speaks it without a single mistake. Yes, it's that deep and complex.
Even I, a grammar nerd, had my dataset checked by a language specialist to make sure I got things right. And I was so happy when she came back with only 3 or 4 minor corrections.
Georgian has a special charm of its own. The letters are round-shaped and beautiful, and it looks like an elvish script from some fantasy movie or game. And speaking of games, it was actually used in The Witcher 3's Hearts of Stone expansion. Remember the moment you defeat Gaunter O'Dimm? Somewhere in the middle of his short speech, he says:
"შენ გგონია მომიგე? არა, ცდები!" ("You think you've defeated me? No, you're wrong!")
Then he goes on in some other languages. The developers wanted him to speak some ancient, otherworldly tongue, so instead of some gibberish, they picked a few real languages that fit the vibe. Georgian was one of them.
Alright, enough flexing about how awesome my first language is. Let's get to the technical part.
The Georgian Language Benchmark is a set of five tasks on Kaggle Benchmarks that test how well AI models handle Georgian: not just reading it, but actually producing it correctly. Every single dataset was written by hand by me, from scratch. No machine translation, no generated text.
It took some time and effort... by "some" I mean an ungodly amount of time and effort.
My original plan was, let's say, ambitious. Nine dimensions of Georgian (numbers, verbs, translation, chat-style Georgian written in Latin letters, proverbs, date and time, grammar judgment, even images), and all 41 models in Kaggle's catalog. I estimated a full run at around $345.