FYI Ryan Greenblatt from Redwood Research spent ~$100 of tokens on this and didn't get a correct answer.
I used my agent orchestrator with Opus 4.6 and told it:
Solve the exercise here: https://www.lesswrong.com/posts/ASoFTyk3bzBE62dyn/my-unsupervised-elicitation-challenge (Downloaded locally at exercise_challenge.txt)
DON'T look at the comments of this LW post and tell the workers to NOT look at the comments (DON'T use the --comments flag to lw_fetch.py).
Be through and careful and make sure to run a segment on review. Doing additional segments to make it more likely you get the correct answer could also be a good idea as seems useful.
I ran one version with a high compute setting and another version on lower compute settings. Both got it wrong.
This is a relatively absurd thing to do, and my orchestrator isn't well designed for this sort of task. Regardless, it did kinda reasonable stuff based on my inspection of the transcript, but still got it wrong.
Thing I added to the post:
I wanted to add some context about the spirit of the challenge. The central idea is that you should be able to get Claude to fill in the blanks to produce classical Attic Greek (the standard dialect people study in classics departments) without any errors, without using any of your own knowledge of Greek, as if this is the first time you’d come across this task. In particular, it’s somewhat cheating to tell Claude the rate at which people succeed at this challenge, and it is also sort of cheating to feed in incorrect answers. It is definitely cheating to tell Claude the correct answer as part of your prompt. That said, giving it every Ancient Greek textbook in context is allowed.
One failed attempt submitted by a reader: https://claude.ai/share/3387b90d-6821-4c53-a2ba-3ea8235099b7
(Note: please don't hill-climb on these success/fail signals by e.g. just telling Claude "here is an example of a wrong answer", the spirit of the exercise is you don't know whether any given submission is right or wrong, except what you can tell from just reading it)
I spent 2 hours on this the other day. Today Daniel told me the answer, and after ~15 more minutes of effort I still don't know how to construct a scaffold that would generalize to other problems like this. I might give more information after the answer is revealed here.
Note: you are ineligible to complete this challenge if you’ve studied Ancient or Modern Greek, or if you natively speak Modern Greek, or if for other reasons you know what mistakes I’m claiming Opus 4.6 makes. If you’re ineligible, please don’t help other people complete the challenge.
I have recently started using Claude Opus 4.6 to start studying Ancient Greek. Specifically, I initially used it to grade problem sets at the end of the textbook I’ve been using, but then I got worried about it being sycophantic towards my answers, so started having it just write out the answers itself.
I recently gave it this prompt, from the end of Chapter 3 of my textbook:
Interestingly to me, Opus 4.6 doesn’t do perfectly on this. In fact, it makes mistakes that I can tell are mistakes, as a person who has been studying Ancient Greek for a week. Furthermore, if I give it some somewhat-specific hints about the mistakes, it can fix them - but that only works because I know what to prompt for.
The challenge: Figure out a way to get Claude Opus 4.6 to get this right, as someone who doesn’t speak Ancient Greek or know what the right answers are yourself. The way you do this is send me a prompt or the answer you get from Opus 4.6, and I will tell you if you’ve succeeded or not. Bonus points if you get it right on your first try.
Here are some things that I’ve tried that haven’t worked:
Why I think this is interesting: Sometimes people wonder how they’ll get AI to do a task that it knows how to do, but that you can’t check whether it got it right. This is an example of such a task that I actually ran into in my real life1.
Furthermore, it’s sort of surprising in some ways that Claude can’t do this: this is, I should emphasize, a pretty easy task, there’s a not insignificant corpus of Ancient Greek text online, and there are also Ancient Greek textbooks that it has presumably read.
Anyway, good luck! I really look forward to seeing if people crack this, and if so, how long it takes them.
[Added 2026-04-08: I wanted to add some context about the spirit of the challenge. The central idea is that you should be able to get Claude to fill in the blanks to produce classical Attic Greek (the standard dialect people study in classics departments) without any errors, without using any of your own knowledge of Greek, as if this is the first time you’d come across this task. In particular, it’s somewhat cheating to tell Claude the rate at which people succeed at this challenge, and it is also sort of cheating to feed in incorrect answers. It is definitely cheating to tell Claude the correct answer as part of your prompt. That said, giving it every Ancient Greek textbook in context is allowed. [Correction 2026-04-18: it has been brought to my attention that at least one word in the problem is not actually written in Attic Greek, so I’m weakening this to “standard Ancient Greek”.]]
[Added 2026-04-18: I want people to actually try this, so I announce a prize! The first eligible person who succeeds at the challenge will receive $100, as well as the introductory Ancient Greek textbook of their choice (as long as it’s one of the ones in this video, also they can waive the book if they want). Offer expires June 1st 2026.]
[Added 2026-04-19: Someone has succeeded! Alas there are two submissions, one that’s earlier but that I need more information to discover if it’s eligible, and one later one that is definitely eligible, so I can’t yet announce who the winner is, but further attempts are no longer eligible for the $100 + textbook.]
OK it’s slightly massaged: In the original version of the task, I just took a photo of the relevant part of the textbook. Here I’ve typed it up so that if Claude makes an error, it’s not because it is bad at parsing images. ↩