Proof Ingredients: Is AI going to replace software developers?

How good is AI at software coding?

Carl Brown, founder of the YouTube channel Internet of Bugs, has made a name for himself by holding AI claims up to scrutiny. So we asked the physicist and software developer to help us assess how good AI is at coding tasks.

Using our AI testing software, which simultaneously queries five leading AI models, Brown asked the models coding questions. He published the results on his YouTube channel and spoke with Proof founder Julia Angwin for our Ingredients video interview series.

Ingredients

Hypothesis: Generative AI cannot replace software engineers, but it can do parts of the job.

Sample size: A dozen questions were asked to five AI models: OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, Google’s Gemini, Mistral’s Mixtral, and Meta’s LLama 2.

Techniques: Posed three types of questions to models: ones that require recent coding knowledge, ones that have multiple solutions, and tasks that require planning.

Key findings: AI models often produced generic answers instead of producing tailored solutions to or plans to execute the specific task at hand, and overall, fell short of what one would expect of a human software engineer.

Limitations: Questions were limited to those that someone who does not code would likely understand. The sample size was small and models may perform differently after updates.

Why we think news needs an ingredients label
   • What's in your news?  

Links

Carl Brown's video about this investigation
   • AI Coding Crap: More Examples. Claude 3.5 ...  

Carl's video debunking Devin
   • Debunking Devin: "First AI Software Engine...  

Carl's YouTube channel, Internet of Bugs
   / @internetofbugs  

https://www.proofnews.org/
  / proof_news  
  / proof__news  

Join us in making trustworthy, verifiable information the new baseline:
https://www.proofnews.org/donate/