DEVOPSdigest

Can AI really write code? Do you even want it to? Can you trust the output? Developers are asking these questions in 2024.

DEVOPSdigest invited experts across the industry — consultants, analysts and vendors — to comment on how AI can support the software development life cycle (SDLC). In Part 7 of this series, the experts discuss the maturity of AI and try to answer the question: Is AI mature enough to actually write code?

In this series, the experts have already shown that AI has many other uses in the SDLC beyond code creation, but in Part 7 we will be focusing on this prominent use case of AI.

Today, AI-powered code creation tools generally fall into two main categories: code completion and code generation, says David DeSanto, Chief Product Officer at GitLab. Both are already effective, however they are continuously improving.

Code Completion

AI code completion is a feature within IDEs and code editors that provides automated suggestions for completing the next few lines of code based on context, prior code, and best practices, DeSanto explains. It speeds up the coding workflow predicting the most likely pieces of code to be written next.

"At this moment, AI's ability to write code on a code-based level or system level isn't very strong," Kai Du, Head of Generative AI at Turing, asserts. "However, if you zoom in to certain languages like Python or Java, and look at a code block level or function level, it's pretty good at autocompleting or completing code. It's still early in the game for AI to write code, but it's pretty useful."

AI is good at writing boilerplate code for common tasks, generating smaller code snippets and programs from natural language, autocompleting code, assisting in troubleshooting errors and helping with code refactoring and optimization, adds Shourabh Rawat, Senior Director, Machine Learning, SymphonyAI.

Code Generation

AI code generation is broader in scope and is common code that is created based on a single-line or multi-line code comment describing the desired outcome. This speeds up development by including complete functions as well as improvements and potential changes to existing code, DeSanto from GitLab says.

"Research shows even today, generative AI coding assistants can increase code output by up to 55%, which is incredible!" exclaims Matt Healy, Director of Product Marketing, Intelligent Automation at Pega.

"The advances in AI's ability to write code have been nothing short of amazing," explains Chris Du Toit, Head of Developer Relations at Gravitee. "Which developer hasn't tried ChatGPT for code generation and not been left suitably impressed?"

Jeff Hollan, Head of Applications and Developer Platform at Snowflake, says, "I would never suggest that someone fully automate software development or leverage AI written code without any approvals. That said, AI is really good at writing code within the context of your app — it knows your variables, it knows the context, and more often than not it runs as is. I think of AI written code today like a super intelligent reference guide right next to you. Instead of looking up docs and remembering 'how do I write this part of code again?,' AI will just suggest it right away."

"Where things break down is in all the other 'stuff' you have to do when developing software, and for which AI and AI agents haven't really broken through," warns Sterling Chin, Senior Developer Advocate at Postman. "For example: create and scaffold an entire github repo from scratch, set up github actions for CI/CD, manage deployment and infra. So there is still a lot for AI/automation to do, but generating code, that pretty much feels like a solved problem today."

Just the Beginning

AI should act as a sidekick to your development team, not a replacement — like a Batman and Robin situation

AI's ability to write code is rapidly advancing, but it's still in its early stages of maturity, and developers should approach it cautiously, advises Dotan Nahum, Head of Developer-First Security at Check Point Software Technologies. While AI-powered code generation tools can assist with various tasks like generating boilerplate code and suggesting code completions, they are not yet capable of fully replacing human developers. Current limitations include a lack of complex problem-solving, limited creativity and innovation, and a remaining need for oversight and validation. In essence, AI should act as a sidekick to your development team, not a replacement — like a Batman and Robin situation.

Many experts would assess AI's ability to write code as entry level, or maybe a little better.

David Brooks, SVP of Evangelism at Copado, says: It is not 100% but it has reached the level of junior developer which is good enough for real use given the productivity gains.

Michael Webster, Principal Software Engineer at CircleCI, says: In my experience, AI's ability to write code is at an advanced beginner to intermediate level. It's heavily dependent on how much context you are able to provide and the patterns the model is being trained on. AI performs well with very specific instructions and established code patterns, particularly if you're able to integrate it with source libraries that enhance its understanding of your codebase.

In a very real sense, AI is at the senior/staff level in terms of book knowledge, but it lacks the wisdom that comes with hands-on experience over many years, Chin from Postman clarifies. It can help guide an engineer to the solution, but not fully understand all the disparate systems. Think about it like this, you've got a new graduate with a CS degree who has memorized every book on programming languages without any hands-on experience. It has limitations, which is why many senior and above level engineers don't believe it's worth learning how to use. However, for the less experienced engineers, it's the perfect tool to get them to the next level.

Marcus Merrell, Principal Test Strategist at Sauce Labs, provides an even starker warning: All AI-generated code should be tested as though it were written by a sophomore in a computer science program.

"The ability of AI to write code sometimes feels uncanny," Ed Charbeneau, Developer Advocate, Principal, at Progress observes. "However, AI still frequently hallucinates providing samples with incorrect syntax, missing variables or deprecated API calls. While experienced developers will quickly identify and correct these mistakes, beginners may encounter roadblocks. It is important to understand that AI-generated code is meant as a starting point and not a ready-to-run solution."

Keeping It Small

For the reasons cited above, many experts seem to agree that AI currently is only mature enough for minor tasks.

AI's best coding use case is writing small, well-defined requests, like generating code snippets or writing routines, says David Brault, Product Marketing Manager at Mendix. However, when it comes to AI's ability to write an entire business application or attempt complex, large-scale projects, AI's capabilities are still limited. Even though AI's ability to write code continues to mature, we are not yet at the point where AI can replace human developers for comprehensive and creative development work. As the scope gets larger, AI's ability to write code becomes less effective.

Randall Degges, Head of Developer & Security Relations at Snyk, adds, "While AI is still nowhere near as capable as a human software engineer, its ability to quickly generate small, limited-scope pieces of working code is excellent and has become indispensable as a standard development tool."

"While AI writing code is a compelling demonstration of its potential, current AI capabilities do not extend to writing or scaling large enterprise-grade applications," Austin Vance, Co-Founder and CEO of Focused Labs, agrees. "Instead, AI can generate standard code within specific contexts or smaller components."

Mathieu Bellon, Senior Product Manager at GitGuardian elaborates: AI currently does not work for complex, time-consuming tasks that involve many moving parts, such as fully integrating the backend and frontend with LDAP or Oauth to payment systems, all while integrating tests. That is the actual day to day job of the developer.

Human Oversight Required

In terms of AI maturity, all the experts agree that human developers are still required for guidance and oversight.

"It is easy to be surprised at the ability of AI to generate code. What you see though is that such code must be carefully reviewed by humans currently to make sure it's doing what is wanted and for possible hallucinations," Arthur Hicken, Chief Evangelist at Parasoft cautions. "It's not mature enough yet to be trusted on its own without supervision."

AI's code-writing capabilities are progressing rapidly. It can generate functional code snippets and provide context-based code completions. However, human oversight remains essential as the generated code may not always be optimal or follow best practices, adds Ramprakash Ramamoorthy, Director of AI Research at ManageEngine.

"While AI tools available today help with writing code faster, they are not perfect," Johannes Dahse, Head of R&D at Sonar agrees. "Multiple academic studies show that state-of-the-art LLMs initially generate code with quality issues, bugs and security vulnerabilities. The technology is still advancing and these tools will continue to be refined. Until then, it will be vital for developers to have quality checks and assurances in place to ensure AI-generated code is up to standards — just like human-generated code."

A Long Way to Go ...

We still have a long way to go on the maturity scale of things when we consider other outliers, but as of now, it is something that the enterprise space can certainly make use of and feel confident they are on the right track, according to Sean Heide, Research Technical Director at Cloud Security Alliance.

"I'm impressed that AI does as well as it does. But I'd have trouble calling AI's code generation ability 'mature.' We're still in the 'eager to please intern' stage," says Mike Loukides, VP of Emerging Tech Content, O'Reilly Media. "There are still a lot of bugs; I don't think I've ever seen an AI generate code using more advanced language features. (For example, Python comprehensions — and comprehensions were introduced in Python 2.) I've spent a lot of time trying to see if AIs could implement the Miller-Rabin prime number testing algorithm, and we've only recently gotten to the point where they can do that successfully."

Loukides continues, "I've also done a number of experiments where I pasted in some code I'd written and asked the AI for suggestions on improving it. I was underwhelmed with the results. The AI never spotted anything that I'd consider a serious weakness. The AI would try gamely (but not always successfully) if I said something like: 'I think this code would be simpler if I used the X function, but I'm not sure how.' If you want AI to coach you on how to improve your code, you shouldn't have to tell it how to improve your code."

Loukides adds, "The bottom line is that the training data isn't all that good. A lot of the code in GitHub isn't all that good. A lot of the code on StackOverflow isn't all that good. Nothing against GitHub or StackOverflow, that's just life in online code repositories. When you have everyone from 6-year-olds to people who have been programming since the 70s and haven't learned much since (that might include me), you get what you get. And once you get beyond Python, C, C++, JavaScript, and Java, the amount of code available to train on gets small very quickly."

AI's ability to generate correct code has definitely improved over the last year, though. That improvement is sure to continue, Loukides concludes.

... But Still Advantageous

"In my experience, the final output of the code is never production-ready, but it cuts way down on research and the time you have to spend reading docs and specs," says Merrell from Sauce Labs.

AI is becoming pretty good at writing code, but it's not perfect yet. It can handle simple tasks like generating boilerplate code or automating small parts of a project, and tools like GitHub Copilot are already helping developers every day, confirms Casey Ciniello, App Builder, Reveal and Slingshot Senior Product Manager, Infragistics. But for more complex projects, AI still needs a guiding hand from a human developer. So, while it's not replacing developers anytime soon, AI is certainly becoming a valuable tool in the toolkit.

Go to: Exploring the Power of AI in Software Development - Part 8: Adoption