AI Coding Assistants Don't Boost Productivity Or Prevent Burnout: Study

While the advancements made in generative artificial intelligence (GenAI) are relatively new and span nearly 2 years (OpenAI’s ChatGPT was launched on November 30, 2022), the infusion of artificial intelligence (AI) isn’t really a new concept.

AI technology and AI tools have been around for a long time and have been used to boost productivity and reduce work pressures for workers by enabling the automation of routine tasks.

One of the sectors AI was majorly deployed in was that of coding. AI was meant to help coders develop and deliver software applications (including the currently controversial SaaS business model).

Have AI Tools Helped Developers?

According to a 2022-published study (the study was conducted in 2021), 88% of software developers surveyed said they felt more productive when using GitHub’s Copilot.

More importantly, 59% felt less frustrated when coding, 60% more fulfilled with their job and 74% felt they were empowered to focus on more satisfying work.

One of the senior software engineers surveyed said, “I have to think less, and when I have to think it’s the fun stuff. It sets off a little spark that makes coding more fun and more efficient.”

As for the parameters regarding efficiency and flow, the readings included faster completion (88%), faster with repetitive work (96%), more in the flow (73%), less time searching (77%) and less mental effort on repetitive tasks (87%).

In a further study that comprised of 95 developers split into two groups, where 45 used the GitHub Copilot and 50 didn’t, it was found that 78% of the Copilot users finished the task of writing a web server in JavaScript, which was completed in 1 hour and 11 minutes (average time).

In contrast, the 50 developers who didn’t use the copilot witnessed a task completion rate of 70%, taking 2 hours and 41 minutes (average time), making it 55% slower than when Copilot was used.

Coming up to 2024, GitHub’s copilot now sports a blend of GitHub's Copilot AI and OpenAI’s GPT-4 to generate intelligent fix suggestions, along with other components bringing a wide range of capabilities. These are said to make it extremely useful for 50.9% of the 165 software engineers surveyed and “Quite a Bit” useful for 29.7%.

However, this is not the case according to an Uplevel study that surveyed 800 software developers.

Source

What Did Uplevel Find?

Uplevel is a firm that analyzes coding metrics and according to the developers in its study, the hype of GenAI and special tools haven’t been able to generate boosts in productivity as yet.

The study, conducted by Uplevel Data Labs, analyzed the difference in how teams with and without Copilot access performed, based on metrics such as pull request (PR) cycle time, throughput, bug rate and extended working hours (“Always On” time).

Where the expectation lay that GenAI and Copilot could enhance coding time and quality leading to lower cycle time, more PRs and fewer bugs without burning out developers, it didn’t.

In fact, developers noticed that code written by Copilot had 41% more bugs.

Furthermore, it didn’t make much difference when it came to PR cycle time, throughput and tests or even coding speed. As such, cycle time decreased only by 1.7 minutes.

More importantly, this AI coding assistant couldn’t help reduce the risk of burnout, according to Uplevel’s “Sustained Always On” metric, which includes extended working time outside of standard hours and is a leading indicator of burnout.

Here, the metric decreased for both groups. However, it reduced 17% for those with Copilot and nearly 28% for those without Copilot.

“Using LLMs to improve your productivity requires both the LLM to be competitive with an actual human in its abilities and the actual user to know how to use the LLM most efficiently,” said Ivan Gekht, the CEO of Gehtsoft. “The LLM does not possess critical thinking, self-awareness, or the ability to think.”

“Software development is 90% brain function — understanding the requirements, designing the system, and considering limitations and restrictions. Converting all this knowledge and understanding into actual code is a simpler part of the job,” Gekht added.

“It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it.”

Has generative AI helped you boost productivity and improve operations? If so, how?

Let us know in the comments below!

First published on Wed, Oct 2, 2024

Enjoyed what you read? Great news – there’s a lot more to explore!

Dive into our content repository of the latest tech news, a diverse range of articles spanning introductory guides, product reviews, trends and more, along with engaging interviews, up-to-date AI blogs and hilarious tech memes!

Also explore our collection of branded insights via informative white papers, enlightening case studies, in-depth reports, educational videos and exciting events and webinars from leading global brands.

Head to the TechDogs homepage to Know Your World of technology today!

Disclaimer - Reference to any specific product, software or entity does not constitute an endorsement or recommendation by TechDogs nor should any data or content published be relied upon. The views expressed by TechDogs' members and guests are their own and their appearance on our site does not imply an endorsement of them or any entity they represent. Views and opinions expressed by TechDogs' Authors are those of the Authors and do not necessarily reflect the view of TechDogs or any of its officials. While we aim to provide valuable and helpful information, some content on TechDogs' site may not have been thoroughly reviewed for every detail or aspect. We encourage users to verify any information independently where necessary.

Tags:

Artificial Intelligence (AI)Emerging Technology Artificial Intelligence AI Generative Artificial Intelligence Generative AI GenAI Burnout Productivity Boost Coding Assistant GitHub Copilot

Join The Discussion

Emerging Technology