Using LLMs to improve Build Process

Context:

While we continue to use LLMs extensively in general best practices like Unit Testing, Validations, documentation etc, we wanted to try to improve our build files. Now, most developers do not get an opportunity to work on build files. Usually the build file for a project is written once by an architect or senior developer and barring some major re-architecture, used 'as is' by the rest of the team for the lifetime of the project. There is not much opportunity (nor need) for someone to work on build files continuously. This stunts growth of expertise in this area.

Our project too had a similar situation. We had a webpack file written a few years back and the only one time that we made any changes was when the project grew large and we had to add caching. This can obviously be seen in our git history.

Given the time and effort required to learn webpack in depth, experiment with it and finally evolve a world class configuration, we kept putting it off for months. Even with our enthusiastic adopotion of genAI (LLMs like chatGPT and Claude Sonnet) for generic software development hygiene, there as always the tension of pausing new features to work on improving the build file.

The experiment:

When we heard of Agentic AI and openDevin OpenHands (openDevin changed their name to OpenHands), we decided to experiment with it to see if that can speed up the process. These experiments are documented here:

1. Using openDevin to build a Typescript (ts) config file for webpack - https://github.com/Digital-Assistant/Digital_Assistant_Client/wiki/Experiment:-Improving-webpack-Configuration-with-openDevin-and-Sonnet-3.5-200k
2. Using openDevin to build a Javascript (js) config file for webpack - https://github.com/Digital-Assistant/Digital_Assistant_Client/wiki/Webpack-Configuration-Refactoring:-Testing-with-a-simple,-unsophisticated-prompt

Some of the key findings:

1. openDevin is quite magical for an early stage Agentic framework.
2. openDevin is not 100% effective standalone.
   1. I started with openDevin to generate the first cut of the config file. I then used Sourcegraph Cody (with sonnet 3.5) and Claude 3 Opus (subscription) to fix some minor issues and some more optimisations.
3. Claude 3 Opus is noticeably superior to Sonnet 3.5 in complex scenarios. Opus correctly suggested that I use ts-node in my package.json to run my webpack.config.ts file while sonnet 3.5 kept asking me to rename the config files to webpack.config.mts or some other wrong alternatives.
4. On the whole, it is astonishing that without even the basic knowledge of typescrip, javascript, react or webpack, these tools helped me accomplish this task at a world class level.