Teaming with ChatGPT as my Developer

The Brief

Use ChatGPT to code a mobile game concept I conceived in 2014 and make it available for the public

Project Ambition

I wanted to explore the limits of ChatGPT as my sole partner in a full design process from concept through product delivery to understand current design tool maturity

Software and Tools

ChatGPT 4 // Xcode // Hand Wire Framing // Figma // Javascript // HMTL & CSS // Adobe Photoshop // Transmit FTP // Personal Web Host

5 Week Build

Role: Product Owner // AI Prompt Engineering // Code Interpreter

Dusting-Off  the Right Concept

Opportunity is Knocking

As I listened to this interview with Sam Altman, I was excited to understand the ways AI might weave into the UX design process.

When I learned about TLDraw a few days later, I realized how large the world of artificial intelligence tools had grown in the last year. This inspired me to attempt to bring some of my early app concepts to life using these new AI Coding resources.

I was captivated by the iPad when it was announced, and spent spare time over the next five years exploring ideas and app concepts that I thought would make compelling products. These design efforts hit a wall when the concept inevitably needed a developer to progress further. 

I spent some time trying to teach myself objective C, later Swift when it was released. Despite huge advances in simplifying the app development cycle, I still found my app concepts too complex for me to piece together on my own. 

As I re-evaluated these ideas, I narrowed them down to 4 concepts, 3 older ideas, and a newer AR-focused app idea just to understand how the AR app design worked. Each started life with the hope of becoming a native iOS app, so they could potentially become commercially viable products, not just example case studies for this portfolio.  I began exploring tools to help  bring one of the concepts above to market using AI to do all of the coding work.

A Look at AI and Modern Development Tools 

I wanted to understand how developers were building AI into their workflows and which of my four concepts would be the best fit for the new tools and workflows

Tool Analysis

My focus was on the product, not on mastering a development tool or process.  I used this as a primary factor in my review of the options

Having a Code Co-pilot

If I had experience with a programming language and best practices, I am sure this tool would be phenomenal. Instead of a tool that would performed turbo-charged auto-completing of my code, I needed a tool that could determine a development structure and completely generate the code, capabilities these tools don’t have.

SAAS App Builders

Online SAAS App Building platforms have proliferated and promise to dramatically reduce the effort to build an app, many tools require no code at all. These require ongoing monthly subscription costs and tie the product owner to a particular SAAS design tool which puts the app at risk of being orphaned in the future.

Flexibility

After reviewing the app builder features and limitations, I realized they would not be flexible enough to achieve the vision I had for each of my four original app concepts. When I saw the video below, I was convinced JavaScript was the right approach, given it powers some of the most advanced web applications, can be tested on any device, is free and open source, and has extensive online support and documentation.

Distribution

My ambition was to release the product of this case study in the App Store to study its potential commercial opportunity. I found the online app builders charged more for app distribution features, reasonable costs for a business, but more than I wanted to pay. When the decision to use JavaScript came into focus, I realized that an online game had far lower barrier-to-entry compared with asking someone to download another app.

Selecting the Development Platform

After reviewing both my existing skillset and my primary objectives for this project, I decided to focus my efforts on the process of making something with AI rather than putting too much commercial focus on its results. 

Instead of investing my time learning a new online app building or coding tool, I wanted to explore ways to leverage the AI superpowers that are available right now.

Once I reframed the objective and settled on the tools, I was excited for the prospects of designing the game with JS, CSS3 and HTML5 as the game’s technology stack.

Setting Project Objectives

I put pen to paper for this game concept on a lunch break in 2014, just a few months shy of 10 years ago. I worked through a few more ideas the next week, but then after trying to program the game in Swift, abandoned it as being too complex to approach as a novice programmer.

As I started work on the concept again, I didn’t have my old notebook of wireframes on hand, so I worked from memory to lay out the general outline of the gameplay. By the end, the final game was quite similar to the original concept, with just a few changes.

I began by identifying the primary variables and features the game would need in order to be a challenging, fun experience. The coding approach was developed over the course of the project in concert with SwiftCoder to address the game’s increasing feature complexity.

Design Iteration

Goal Statement

I will create a functional web-based game using SwiftCoder to write the code in JavaScript (a language I have no experience with) and leverage my HTML and CSS experience to guide the prompts and work through code integration.

Beginning with an AI Persona

The logo SwiftCoder gave itself

I developed my GPT persona as SwiftCoder: An  Expert in crafting optimized Swift, HTML, JavaScript code with clear explanations. 

I started with ChatGPT 3.5 and while impressive, it was access to ChatGPT 4 that truly showed the power of AI as an insightful coding partner.

The “Ideal” Coding Workflow Emerged

SwiftCoder’s achilles heel is maintaining specific context when tackling complex feature requests

I started with zero experience with AI chat bots or technologies and found Chat GPT 3.5 as impressive as I had been reading about. I also found its self-confidence was far higher than its skill level warranted.  I knew to look out for this, but found it interesting to see in action.

I generally do not feel comfortable handing over creative writing, or other fact-based writing assignments to AI, given the very real problem of AI hallucinations.

I didn’t have this reservation when using AI to program; by using Javascript validators like www.CodePen.io and having the ability to quickly test code, I had confidence that together we could successfully complete the project.

To be completely honest, I was also immediately swayed by the prospect of finally developing code for my concepts; a skillset I have longed for, but not devoted the time to master.

I knew from my research that specificity and clear language were important when writing prompts for SwiftCoder. I wanted to push the AI model to see how much it was capable of understanding and responding to correctly.

I found this approach gave SwiftCoder far too much credit and I later found even small portions of the initial requests were challenging and nuanced features that later took days of effort to write, and debug, not something that can just be defined by a few sentences.

As I wrestled with SwiftCoder and felt I was getting diminishing returns, I would abandon the chat and code, and start again with a modified approach, incorporating what I learned from each failure before.

By my third fresh start, SwiftCoder got me to the point where I had an interactive shape on the screen and it felt like we could do anything!  From that point, until the very last prompt, I was glued to the screen, eagerly collaborating with SwiftCoder, providing feedback and testing gameplay.  

The Rocky Road to AI-Coded Success

13 major versions – with 94 sub-versions, saved before significant code integrations and over 3,600 lines of code

You can play many of the versions to get a sense for how things progressed as new features were added.

As the development pace speed up, I implemented a simple version control system by saving the files before major code integration updates. 

I chose this approach instead of investing time to learn a new coding-specific tool such as GitHub, in order to simplify the workflow to its bare essentials. I made a clear distinction that I would not invest energy to learn to develop, as much as I wanted to learn how to leverage AI so I didn’t have to learn anything about that, instead relying on the AI for guidance.

It turns out, my experience working in HTML and CSS dramatically helped the code-integration work, as it gave me a framework of understanding for some basic coding approaches and conventions. If I had no previous HTML or CSS experience, this would have been a much more challenging endeavor. 

Below you can explore an interactive timeline highlighting major stages of game development over the five weeks. 

Timeline

Version 1: Creating SwiftCoder

The foundation for my Coding Partner


This is the foundation for the Coding Partner I was building.  I was able to use ChatGPT 3.5 to set up SwiftCoder, while I waited for access to the more advanced GPT4 model.


I embarked on the game design to see how far we could take things with the LLM that ignited the world’s passion for AI.

Version 3: First Interactive Code

Our First Interactive Screen


Play this Version

 

After trying a few different starting approaches with the prompts, I got a box I could drag around the screen. Oh boy!

 

I used the analogy of a tabletop air hockey game, rather than billiards, I also fine-tuned the follow up information and provided opportunities to get questions back from SwiftCoder.

Version 7: First ChatGPT4 Code

Almost like Wonka’s Golden Ticket


I was able to use ChatGPT 3.5 to set up SwiftCoder, while I waited for access to the more advanced GPT4 model. I embarked on the game design to see how far we could take things with the LLM that ignited the world’s passion for AI.


It turns out the new model was a world apart from the previous generation and the coding immediately began to feel more capable and I had a much better guide when I didn’t understand the instructions, or how new code was being combined with older code.


The game from this version didn’t work, ultimately, but I learned a lot about how to approach this new model effectively.

Version 8.14: Messages for Game Status

The first functional user affordance

 

Play this Version

 

The initial game messaging was provided in console logging, I told SwiftCoder messages need to be visible to users in the web browser.  The next option that it coded used a modal pop-up, which is nice, but doesn’t allow future CSS styling and layout control. It took several rounds of updates before we got to a place where we could trigger different messages for different events.

 

Once I had determined the ways to time the message display to key moments in the gameplay, I had to work hard to maintain the code.  As new game message triggers were added, the conditional logic became bloated and required the first big code refactoring.

Version 9: SwiftCoder is Self-Aware

SwiftCoder Provided this Unsolicited Self Portrait


I was kind of shocked when I asked SwiftCoder to consider an approach to fixing a persistent attempt-counting issue that left the game non-functional.

 

Its response started with the image to the left, followed by an insightful set of strategies to solve the behavior, including some new debugging console messages to track the progress of the game.

 

While it provided great solutions, I found it interesting to consider how SwiftCoder chose to express its answer by first humanizing the response.

Version 10.13: Quality Control on the Road

Fixing attempt counting bugs

 

Play this Version

 

Thanks to 5G cell service I spent 4 hours debugging the game while we drove to and from a family birthday.   I was able to focus on solving a handful of persistent bugs that had been impacting the gameplay.

Version 11: Live on the Web

Finally publishing beyond my Mac


Play this Version

 

Just in time for my daughter’s 5th birthday, I got a version on the web so she could play it as a gift.  The first outside user-testing is an eye opener, for sure!

 

It was gratifying to see her and her brother having fun, but it gave me a strong incentive to improve the user experience beyond its basic functionality.

Version 13.2: The Styling (Finally) Begins

Starting to test some styling


Play this Version

 

I wanted to begin to shape the user experience and set a visual hierarchy for the different elements in the UI.  Up to this point, I had been building the UI in a modular fashion to allow for flexible layouts, easy styling and logical control structure for the user.

Version 13.5: We have a Scoring System!

The Scoring System is Live

Play this Version

 

I spent a morning working with SwiftCoder to implement a strategy to I had to create a simple way to generate a score.  I would use remaining attempts as a base for the score, with one point per attempt remaining. Then, I used the number of bounces the player needed to generate a “bounce bonus” giving 2 points for each bounce they had remaining.

 

Like most new features, this one took a little while to ensure the scoring functions properly counted the bonus bounces. It was incredible how effective troubleshooting with SwiftCoder was, considering I was observant, and could describe what was happening in the game, but had almost no input regarding the code and through some engaging back-and-forth, we were able to resolve some really sneaky behavior.

Version 13.23: Slingshot Dynamics
 

Unlocking a new Experience


Play this Version


Given difficulties using the drag-to-play dynamics on touchscreens, I was curious how difficult this would be, and realized given my expert coding assistant, it didn’t take more than a few prompts to get close to a working solution.


Then began the in-depth efforts to get it to show up consistently and across all levels.  I found it challenging to find the solution without quite a bit of discussion with SwiftCoder.

Version 13.35: The Game is Feature-Complete

The Game is On!


Play the Game

 

The game has finally reached the stage of being feature-complete and playable across all devices. The final steps are to update the styling to its final designed-state. Through the various iterations of this project, it has gained dozens of features, when measured at their base, programmatic building blocks.

 

I will never take for granted a request to “just display some messages when different things happen in the game” for what it truly entails to put in place.

Testing the Gameplay

Testing was crucial after every code update and feature added. A methodically incremental approach for adding features was most effective, but this later lead to large efforts to refactor the code as bugs and inconsistencies hampered progress

SwiftCoder was instrumental in providing meaningful troubleshooting strategies and code validation at every stage of the development process


As the project started, I had little understanding of the JS code being written and I would copy paste the entirety of a file’s code back into the chat with a description of the current, errant behavior. 

Later as I became more familiar with the code base and how some of the primary functions were organized, I was able to send shorter code sections back for review. This lead to fewer AI-introduced errors, and fewer human-introduced copy errors when integrating code.

This didn’t mean troubleshooting was easy, or getting the results that seemed straightforward were actually straightforward. I found myself continuously impressed with how a simple thing like getting proper game reset to happen was nearly impossible, while implementing a scoring system took only half a day. This probably speaks to my expectations and understanding of development more than anything.

Refining the Experience

The game received consistent gameplay from my two kids as soon as it was available online. I conducted brief observations while they played to see how approachable the game was for 5 and 7 year olds, without giving them much instruction. 

It was interesting to watch one player who can read and another who can’t, to highlight the need for clear visual cues for primary game actions like goals scored, when the game is over and more. One challenging aspect for both kids was entering initials after getting a high score. In its first version, it would present a typical text box, but was expecting only 3 letters. Users had to press the caps lock key, or hold shift, and there was no check to see if initials had been entered before the submit score button was pressed. This seemed easy enough to the designer, but other users were forced into a multi-step experience just to enter their initials. 

SwiftCoder and I used a handful of incremental improvements to take that ambiguous text field first written into the game and revise it so it would verify the field has the required 3 characters before activating the submit button. 

The final step  was to split the single field into three character fields provides additional clarity to players about the expected input. The game automatically preselects the first field when the form is presented, and the letters are automatically formatted as CAPS to provide visual consistency in the leaderboard.

Design Iteration

The game uses a few simple tools to provide context to the user.  Because I am using as few additional tools and textures as possible, I rely primarily on color to denote materiality. 

Similar the game messaging needed to be presented to the user with more impact, to help players who may not be able to read yet, or who speak another language.  Using messaging colors also provides simple feedback about game status.

During the development cycle, I recalibrated my expectations for what seemed like a simple feature like having a resizable game board requires extensive coordination across the code base. 

Accomplished in stages, this critical feature started with a fixed 1024×768 game board, then added the ability for the game board to resize to accommodate the dashboard height using a fixed ratio. 

The final step was to stretch the game board to the edges of the web browser window, minus the dashboard height. The game resizes dynamically, constantly redrawing the game board.

The triangular wall code was completely rewritten to define the “peak” using coordinates set by percentage of width and percentage of height, making them completely responsive to browser size. The game piece and goal are also sized using percentage of game board dimensions, and they are located using similar coordinate methods. 

This allow the game to be played on any screen proportion, and allows player 

Where can we go next?

Wow. I have wanted to see this game come to life for a decade, and I did it in a month. This process gave me huge respect for the improvements made between OpenAI’s v3.5 Turbo and their v4 model.

I am excited to see how future updates progress and how individuals and companies will continue to integrate AI technologies into their workflows. 

Play on ANY surface with ANY web-enabled VR system

I was deeply inspired first by the iPad, and now we are about to experience Apple Vision in just weeks. As I developed and played this game, I found myself wanting to play like a tabletop arcade game, using ARKit to overlay the game on any table in view.

Practical Applications

Now that I have picked up some basic JavaScript code understanding, I would like to take these skills and develop a native Figma Plugin to help generate useful Persona’s. The great thing about Figma Plugins is they use the same underlying web technologies as the game I just wrote. I can leverage vanilla HMTL, CSS and JS along with a custom Figma API to allow the plugin to interact with the design canvas. 

0:00
0:00