Daniel Paul O'Donnell

Forward to Navigation

Working on code with ChatGPT: Some tips (some also useful more generally)

As a Christmas project, I’m working on a plugin for Obsidian that will convert BibLaTex exports from Zenodo into markdown files that can be used as literature notes. The details of this don’t matter, as the tips I’m going to list here are more generic. And no doubt I’ll write about the plugin once it’s done.

Thus far, I’ve tended to use ChatGPT for a couple of simple kind of coding tasks: roughing in spreadsheet formulas to do specific things in Sheets or Excel, and roughing in small code, CSS snippets, or javascript snippets or developing python scripts. Mostly these work o.k. (certainly it’s been getting better) and even when they don’t, the main thing is I’m in the rough ballpark and have a formula that I can then adapt by hand until it works.

This plugin is the first complex thing I’m building with ChatGPT. I’ve done coding projects like this in the past by hand, but, as a rule, I’m always a novice. With the exception maybe of XML — I used to be Chair of the Text Encoding Initiative, after all — I don’t really work very regularly with any language and so I’m always hacking things together from StackOverflow and so on. So working with ChatGPT to code something big in Java is not really that different from how I used to work pre-LLM. I’ve always hacked things together from code fragments found somewhere. The difference this time is I have an engine that is producing the fragments for me.

Unfortunately, as I understand many students have discovered, ChatGPT is not something that can actually code complex projects through multiple iterations all by itself. As a rule it gets you close, but then as you iterate, it tends to get worse and worse (something I think may happen due to the problems it has when it is fed LLM generated texts more generally).

Lesson 1: Don’t ask ChatGPT to produce entire code files for you

What I’ve discovered doing this project is that a big part of it is that ChatGPT is extremely bad at locking code down. That is to say, when you ask it to fix one part of code it has produced — even if you tell it to leave everything else alone — it very often rewrites other parts of the code, often returning to its original suggestions. So in this case, for example, it reverted to a BibTex parser that doesn’t work (after we’d spend several iterations getting one that did) as soon as I asked it to fix something else. And we had a complex author logic that it abandoned the second I asked it to look at keywords.

What I had been doing that was causing this was asking ChatGPT to reproduce my entire file each time it made suggestions, rather than let me hunt through the code looking for the thing that needed to be fixed. I thought that this would avoid me making mistakes implementing its proposed correction by asking the bot to put them in the right place; but because it rewrites (rather than edits) the code every time, what was actually happening was it was reverting earlier changes it had forgotten about, breaking things in the process (nb. this seems to be less of a problem with o1 than 4o, but even there, I was strict about not letting it rewrite everything).

So lesson 1: Ask ChatGPT for the minimum code required to fix a problem and add it to the code yourself. Do not ask it to put the code in the file itself, as it will breaksomething else

Lesson 2: Ask ChatGPT to always end responses with a response number, date-time, and chat name

I’ve always found it awkward in ChatGPT to refer back to something it said or did earlier, or an earlier prompt that I had given (“could you repeat what you did three responses ago, except this time…” or “Could you go through the chat and collect all the times you said x…”). This is even worse when you are coding.

I’ve come up with a solution that I love so much, I’ve asked the bot to put it in its memory and always use when responding to me: append a response number and date-time-TZ and chat name stamp to the end of every response. So now it ends every response to me something like this:

Etiam sodales lacus commodo, sollicitudin quam sed, aliquam nibh. Sed.

[Response 023 | 2024-12-25 23:11:05 UTC | Chat name: Adding date stamps]

This has turned out to be an amazing improvement. It allows me to reference specific answers and prompts (e.g. “in the prompt before Response 023, I asked…”). But even better, it seems to focus ChatGPT’s attention as well if you ask it to review a chat: so far, at least, it breaks things down by response number (e.g. “I proposed x in Response 019, and added z in response 021” and so on).

Lesson 3: Use Github to ensure that you are tracking versions

Because it messes things up, you really need to have versions so that you can roll things back if your code stops working. If you don’t, you’ll find it impossible to roll things back. Putting them in github means that you have control: you can start new branches, return to previous steps, and so on. I just had a case today where after going through a whole bunch of changes, I decided I needed to roll things back about 30 responses. I don’t what to think what would have been involved in trying to piece together what the code had looked like then. By working with github as well, it meant that I could easily go back.

Lesson 4: let ChatGPT write your commit statements (and include the response datestamp).

I’ve always been bad at writing commit statements, so I asked ChatGPT to give me one every time it proposes a code change… and to preface it with the response date stamp and response number. This has turned out to be amazing, since it means if you have to roll your github back, you can easily find the correct commit, because ChatGPT has included them all in its responses. So once you find that you need to go back to response 020, for example, then all you need to do is search the git log for the commit statement that began “Response 020…”

Because ChatGPT doesn’t get bored, the commit statements are also much better than anything I’d have produced!

 

Comment