First Month of using Github Copilot

Recently was given access to Github Copilot as a trial at work, I had only briefly messed around with other AI Chatbots which I found didn't always give the correct information, which made me a bit skeptical about using one at work, but I also wanted to see what it could do.

I decided to also set up the trial on my personal GitHub to use in my side projects, which allowed me to test it in more scenarios, this is where I found it to be more useful but also weird issues when using it.

A lot of the side projects I have contain a lot of mock data made up of a number of JSON files which is to quickly get a prototype of the frontend app, however when adding a new feature or property this can become annoying on large projects, this where I found copilot to help.

In VS Code I was able to open the Prompt window and ask it to update the data with this property and this data, for example, I used it to add an image from the Unsplash photo library with a search query and it was able to do this to the items in the array. This meant rather me wasting time on updating mock data, I could get the copilot to do it while I focused on the parts I wanted to.

This also translated well to Unit Testing with Jest and React Testing Library, a lot of the time copilot was able to provide an example/skeleton of how to write specs based on the previous tests but with the new test condition, this meant I had a good starting point in unit testing, however there were occasionally issues which I explain later.

The Copilot Chat VS code extension is where I found I used Copilot the most outside of testing and mocked data, this meant that I could ask it questions on how to do stuff without it changing the file I had open.

This was great because it meant I didn't need to leave VS Code and open multiple Chrome tabs when I had a question or an issue and avoid possible distraction, I could simply ask copilot stuff like What does this regex match inside VS Code or suggest a way to do something which I can then adapt to my code rather copilot dropping it in.

This meant I stayed focused on the task at hand, however as with every AI chatbot it did sometimes get it wrong, which did cause me to slow down.

Sometimes I found it slowed down my development of a feature, for different reasons, an example of this was it gave me a misleading answer where it faked a bad path I needed which made me go in the wrong direction when coding.

This made me test out Copilot chat a bit more, where I asked it for some prompts such as 'Write me a function which detects if the day ends with y', which of course doesn't have a bad path as all days of the weekend in Y, yet copilot showed me an example of the function being used and returning false for Monday, Wednesday and another day, this is a very basic example of where it can mislead you if it can't give a proper definite answer.

A lot of the projects I work on are Javascript / Typescript based, which I did find Copilot will sometimes return logic with lots of code smells, or not the quality for production I needed, which is a part of the reason I mainly use the Chat features, allowing me to translate the output to the logic I needed.

One of the most interesting things I did see with using copilot was when I was working on a prototype for scalable GTM/analytics events in a react project, at the time I was near abandoning and looking at other solutions, so I decided to ask the chat tool how it would recommend building what I was trying to achieve.

Its response came back with 3 recommendations, one of which mirrored my own prototype line by line even though it was not finished, this made me worry this could lead to developers getting confirmation bias or believing what they have written is correct when it isn't.

Obviously, like most tools, there is a learning curve to using them to make them more effective, I did find from the initial prompt I gave it when I started at the end of the month had greatly improved, and I was giving more details, as well as me using the chat to be more effective.

As I keep using the tool I am experimenting with the prompts and learning better ways to phrase the prompts, and where copilot's strengths are.

Copilot can be an amazing tool but sometimes I found it could slow me down, and sometimes mislead me, which made me anxious about how other developers might accept the answers it gives them because the AI gave them, and playing more with the tool grew this concern.

I think it has the potential to be a valuable tool for developers to speed up development, but I think it needs time to grow as a tool, as well as teach developers how and when to use it, write good prompts, understand its code output and know its strengths and weaknesses.