AI Impact Research

Comprehensive research on AI platform usage from frontier labs (2025)

View the Project on GitHub vishalsachdev/ai-impact

Back to Overview Data & Analysis →

Coding and Research Capabilities: Sources

SWE-bench Sources

Primary Benchmark

  1. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
    • Type: Academic benchmark
    • Key Data: Methodology and baseline results
  2. SWE-bench Verified Leaderboard
    • Type: Ongoing evaluation
    • Key Data: Current model rankings

Model Performance Reports

  1. Claude Opus 4.5 Announcement
    • URL: https://www.anthropic.com/news/claude-opus-4-5
    • Published: November 2025
    • Key Data: 80.9% SWE-bench, exceeds best human
  2. VentureBeat: Claude Opus 4.5 Analysis
    • URL: https://venturebeat.com/ai/anthropics-claude-opus-4-5-is-here-cheaper-ai-infinite-chats-and-coding
    • Published: November 2025
    • Key Data: Human comparison details
  3. ByteIota: Claude Breaks SWE-bench Record
    • URL: https://byteiota.com/claude-opus-4-5-breaks-80-swe-bench-first-ai-to-beat-humans/
    • Key Data: First AI to beat humans
  4. InfoQ: Claude Opus 4.1 Release
    • URL: https://www.infoq.com/news/2025/08/anthropic-claude-opus-4-1/
    • Published: August 2025
    • Key Data: 74.5% SWE-bench

Productivity Research

METR Study

  1. METR: AI Coding Assistant Impact on Developers
    • Type: Research study (July 2025)
    • Key Finding: Experienced developers 19% slower with AI

Microsoft Research

  1. New Employee Copilot Usage Study
    • URL: https://www.microsoft.com/en-us/research/publication/new-employee-copilot-usage-insights-into-productivity-and-socialization/
    • PDF: https://www.microsoft.com/en-us/research/wp-content/uploads/2025/04/New-employee-Copilot-usage.pdf
    • Published: April 2025
    • Sample: 125 Microsoft interns
  2. Copilot’s Earliest Users Research
    • URL: https://www.microsoft.com/en-us/worklab/work-trend-index/copilots-earliest-users-teach-us-about-generative-ai-at-work
    • Key Data: 70% more productive, 29% faster

Stanford/BetterUp Research

  1. Workslop Study
    • Published: September 2025
    • Key Finding: 40% encounter low-quality AI output
    • Key Data: $186/employee/month productivity loss

Developer Tool Analysis

Coding Tool Comparisons

  1. Index.dev: ChatGPT vs Claude for Coding
    • URL: https://www.index.dev/blog/chatgpt-vs-claude-for-coding
    • Key Data: Feature and capability comparison
  2. GoCodeo: Claude vs ChatGPT for Coding 2025
    • URL: https://www.gocodeo.com/post/claude-vs-chatgpt-for-coding-which-ai-dev-assistant-performs-better-in-2025
    • Key Data: Performance benchmarks
  3. Descope: Developer’s Guide to AI Coding Tools
    • URL: https://www.descope.com/blog/post/claude-vs-chatgpt
    • Key Data: Use case recommendations
  4. ClickUp: Claude vs ChatGPT for Coding
    • URL: https://clickup.com/blog/claude-vs-chatgpt-for-coding/
    • Key Data: Feature comparison
  5. Level Up Coding: Why Devs Switched to Claude
    • URL: https://levelup.gitconnected.com/why-i-ditched-chatgpt-for-claude-ai-and-all-my-dev-friends-are-doing-it-too-e2a1cbeb138a
    • Published: March 2025
    • Key Data: Developer preferences

AI Research Capabilities

  1. Anthropic: How AI Is Transforming Work at Anthropic
    • URL: https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic
    • Key Data: Internal automation patterns
  2. eWeek: Anthropic Economic Index - Coding Dominates
    • URL: https://www.eweek.com/news/anthropic-economic-index-claude-ai-usage/
    • Key Data: 36% of Claude usage is coding
  3. Inc: Claude’s Killer Use Case
    • URL: https://www.inc.com/ben-sherry/anthropics-claude-ai-has-1-killer-use-case-according-to-new-data/91240506
    • Key Data: Software development dominance

CS Education Research

  1. Anthropic Education Report: University Students
    • URL: https://www.anthropic.com/news/anthropic-education-report-how-university-students-use-claude
    • Published: April 2025
    • Key Data: Student coding patterns
  2. Computing at School: AI in Education
    • URL: https://www.computingatschool.org.uk/forum-news-blogs/2025/april/empowering-educators-insights-from-anthropic-s-report-on-claude-s-role-in-higher-education/
    • Published: April 2025

Source Categories

Category Count
SWE-bench/benchmarks 6
Productivity research 4
Developer tools 5
AI research capabilities 3
CS education 2
Total 20