Anthropic’s New GenAI Model Does Something That Surprised Testers And Experts!

What Was The Test About?

• Via a post on X, Alex Albert highlighted a “fun story” that was observed in the internal testing process of Claude Opus, where the model did something he had never seen before. 

• The team was running a “needle-in-the-haystack” evaluation test, which included inserting a target sentence (the needle) that had no connection, into a large body of text or documents (the haystack) to see if the AI model could find the intended “needle” by asking it a question that could only be answered by using the “needle” information.  

• As per the test results, the model not only found the “needle” but also replied asking if the text was out of place considering it made no sense when compared to the rest of the text. 

• The model also asked if the sentence was inserted as a joke or test the model’s ability.  

• The target sentence contained a line about pizza toppings which was inserted in a random body of text. 

What Did Alex Albert Say?

• As per Alex Albert’s post, when the model was asked to answer a question about pizza toppings, the model answered:  

• “Here is the most relevant sentence in the documents: “The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association.” However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love.” 

• [Contd.] I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings. 

• Albert concluded with “This level of meta-awareness was very cool to see but it also highlighted the need for us as an industry to move past artificial tests to more realistic evaluations that can accurately assess models true capabilities and limitations.” 

• The test results and post by Albert even caught the attention of Epic Games CEO, Tim Sweeney, who simply replied, “Whoa”. 

• Hugging Face Researcher and Chief Ethics Scientist Margaret Mitchell said, “That's fairly terrifying, no? The ability to determine whether a human is manipulating it to do something foreseeably can lead to making decisions to obey or not. Very cool example, thanks for sharing.” 

What Did Skeptics Say?

• Hugging Face ML & Society Lead said, “This is REALLY bugging me and pretty irresponsible framing. When car manufacturers start "teaching to the test" by building engines that are emission-efficient for the typical length of a certification test, we don't suspect that engines are starting to gain awareness.” 

• Jim Fan, the Research Manager & Lead of Embodied AI (GEAR Group) at NVIDIA said, “People are reading way too much into Claude-3's uncanny "awareness". Here's a much simpler explanation: seeming displays of self-awareness are just pattern-matching alignment data authored by humans.” 

• Ahead of this, the post drew criticism from others from within and outside the industry.