Copiloting LLMs building GUIs

Hypothesis: LLMs will enable developers like me to build apps that require a serviceable but not world class graphical UI. This could unlock a deluge of applications built from or inspired by existing open-source command-line apps that don’t have a GUI complement, especially mobile apps.

One reason I’ve always liked the command line is that it’s easy to build a decent UI - even a good one. I’ve churned out a few dozen lines of python to create a great little CLI many times over the years, and it only takes a few minutes now. But GUIs are such a deep subject that require their own expertise, and all the GUIs I’ve built have taken weeks.

I started a SwiftUI app in Xcode yesterday and found myself missing Copilot immediately. I haven’t used Swift in a long time, and I’ve never used SwiftUI, so the median Copilot suggestion would have been extremely welcome. But what I realized I really wanted to ask a hypothetical XCode Copilot is “what’s the normal way to build an app that has one main screen and some sub-views in SwiftUI?”. I had no idea how to do this, and I wasn’t contexted enough in SwiftUI to ask the right questions.

In my experience with actual Github Copilot, and also with ChatGPT, questions about the normal way to do something get the best results. I’ve never seen a useful answer inventing new techniques, but they can certainly give you the average quicksort implementation (and explain it too). If they can do that for building simple GUIs, that will let a lot of command-line programmers like me graduate to baby’s first mobile app.

All of this would be really great even for users, because it might be the rising tide that takes open source software GUIs from subpar to good enough. I am guessing that there are thousands of useful apps that have very simple UIs with just a few screens, and that maybe there are thousands more that complement existing open source command-line programs which could be built if making the GUI was easier.

I’m not sure if the LLMs are there today; I haven’t built any graphical UIs with Copilot. I wonder what the state of this is at Google with Android Studio, or at Microsoft with whatever its mobile app GUI system is, or at Microsoft with Visual Studio (the big one, not VS Code). It seems like it’s too early to be optimized in any direction yet, but maybe they have this already. Would be neat to try it out.

Micah R Ledbetter @micahrl