ep.2: HOLL·AI·WOOD AI Film and AI Games Festival, Tackling Gen AI’s User Engagement Problem, Design Research Job Roundup
ep.2: HOLL·AI·WOOD AI Film and AI Games Festival, Tackling Gen AI’s User Engagement Problem, Design Research Job Roundup
8 min read
The second annual HOLL·AI·WOOD AI Film and AI Games Festival took place in San Francisco on October 10. In one year, the festival went from showing AI film experiments to short films and game experiments, with its sights set on feature films and gameplay demos for 2024. Of the twelve pieces shown, my highlights included:
Mors Machina, a self-referential speculative design short film that showed the dark outcome of endless self-improvement via technology.
Looking Glass, a first-person shooter horror game, in which non-player characters (NPCs) are capable of taking actions for and against players in the real world, and autonomous orders of weapons and a favorite latte prepare the player for confrontation.
These pieces also highlighted three takeaways from the event.
Edward Saatchi, CEO of Fable Studios and an organizer of the event, introduces the second annual AI Film and Games Festival.
The three takeaways
Artistry over tech demos: We can build all the text-to-3D models that we want, but we still need strong narrative progression to create quality long-form media. I saw some of that narrative in Mors Machina, which was both thought-provoking and emotionally evocative. Per Edward Saatchi, CEO of Fable Studios and an organizer of the event, “Operate like artists…if the tool doesn't give you what you need, you use something else. That is art." This parallels the conversation in generative AI product development, about building user-first rather than tech-first.
Novel game mechanics grounded in emergent interaction between player and AI call for rethinking game design principles: AI-based games enable new types of social problem solving, like a player training generative agents with different skills to maximize use of scarce resources (here’s an example of this tech coming from Stanford). We’ll also see emergent items appear in the game based on the player’s interactions that can directly alter the game-play narrative. These developments present new opportunities for game designers to extend and develop new game design paradigms.
Creator tooling needs to balance surprising outcomes with creator control: Katya Alexander, President of AI studio Pillars, described how directing horror short “The Red” using AI tooling was a double-edged sword - it would take hours to generate a usable shot, but then a shot would emerge that would provide a radically new perspective, making the hours of experimentation worth it. To achieve “artistry over demos”, we need to prioritize creator control - a new take on the classic interface usability heuristic, “user control and freedom”.
Human-Computer Interaction News
Highlight: Generative AI’s Act Two - Sequoia Report
While ChatGPT remains the fastest-growing consumer application in history (100MM users in about 6 weeks), this report demonstrates that generative AI has a user engagement problem.
Graph from Sequoia’s Generative AI’s Act Two report, showing the DAU/MAU for incumbents and AI-first companies.
The median DAU/MAU for incumbents like YouTube and TikTok is 51%. Compare this to AI-first companies, with a median of 14%. Character AI is a notable exception at 41% - ahead of Duolingo, Candy Crush, and Tinder, and on par with Roblox. In short, while use cases and demand for generative AI is high, most people aren’t yet finding sufficient value to use generative AI daily, relative to incumbents.
What will make Generative AI products something people use every day? An interface to help people express what they want is a start.
Character AI’s high user engagement, relative to other AI-first tools, makes it a great starting place to investigate this question. I did a quick test of Character AI versus the fastest-growing consumer app in history, ChatGPT (v3.5, aka the free version - therefore a good representation of what a majority of users can access). Character AI helps guide people to articulate what they want to do while leveraging the familiar mental model of chatting with a real person, whereas ChatGPT requires people to come to the tool with a clear goal in mind. Given that people generally are bad at coherently expressing what they want (see last week’s newsletter), Character AI is at an advantage over ChatGPT, from a user interface perspective.
The importance of eliciting goals
Opening Character AI’s home page, I notice it is organized by categories of characters you might want to talk with, such as “helpers” and “famous people”. Once you select a category, you see a range of options you didn’t know you wanted. For example, selecting “helpers” displays everything from therapists and fitness coaches, to DJs and “Blender God” to help you learn the 3D tool, Blender. I selected DJ-Next, a “musical recommendation expert”.
A list of Helpers on character.ai
DJ-Next started by asking me a question, rather than me making the first move, as in ChatGPT. This helped me figure out what my goal actually was, rather than opening the tool, already with a clear goal in mind - again, something humans struggle with, and something ChatGPT requires us to do.
Compare how Character AI starts by asking me a question about my musical preferences, rather than me coming to ChatGPT with a specific question about a musical recommendation.
Why helping people elicit their goals leads to greater “variable reward”, and consequently, engagement
Both Character AI and ChatGPT utilize natural language processing to create humanlike conversational dialogue, and therefore leverage the familiar mental model of talking with a person. However, unlike ChatGPT, Character AI also helps elicit our goals through its user interface design. We can map this process to the Hook Canvas, a framework for building user engagement.
The way Character AI helps users elicit their goals increases “variable reward”, increasing the chances of return use.
The “external trigger” for Character AI is more general than for ChatGPT, likely “to have a conversation”, rather than “to get an answer to a specific question”. “Action” is selecting and interacting with a character of your choosing in Character AI. The “variable reward” is the character’s answer to my response. “Investment” is the conversation that I can return to, even after closing and reopening the DJ-Next dialogue.
Putting this all together, Character AI serves as an example of strong user engagement with an AI tool. Something it’s doing right is offering a user interface that elicits people’s goals, helping people both discover and express what they want. How might we use this as inspiration for other generative AI interfaces?
Do you want to chat about how to design your generative AI interface to boost user engagement? We’d love to work with you. Reach out at hello@sendfull.com
Back to our regular programming…
How UX Research Can Meet the Moment: David Lick, User Research Manager at Google, discusses how we’re living through two transformations in technology: the introduction and deployment of GenAI models that have the capacity to profoundly change human productivity, and a renegotiation design research’s role in product development. David offers examples of research questions and methods applicable before, during and after GenAI feature development. He also highlights that many design researchers already have key skills to help scale a human evaluation program (i.e., the gold standard for model evaluation), and offers best practices for doing so.
Use case framework for GPT-4V: Greg Kamradt analyzed 100+ use cases for GPT-4V to build a framework consisting of seven categories. While useful to gain an overview of what GPT-4V can help people do, there may be additional categories like temporal reasoning, as Tom Bielecki points out. That is, GPT-4V can try to understand what happened between two frames, and then try to predict what will happen next.
Greg Kamradt’s use case framework for GPT-4V.
State of AI Report 2023: AI investors Nathan Benaich and the Air Street Capital team released a State of AI Report, analyzing key AI developments. Among many takeaways, they highlight, “challenges mount in evaluating state of the art models, as standard LLMs often struggle with robustness.” As we covered last week, this is further evidence supporting that we need a framework for developing model evaluation approaches. Evaluating if you’re receiving meaningful, safe, or comparable results from one model versus another is currently highly subjective.
AI Is Becoming a Band-Aid over Bad, Broken Tech Industry Design Choices: In this Scientific American opinion piece, Ed Zitron argues that the tech industry’s pursuit of revenue and customer expansion has overshadowed product usability. For instance, to find something in your “garbage dump of apps and options”, you must use Apple Spotlight. AI built to “guide” a user will almost always prioritize the experience that the company wants you to have over the one that you’d actually like (e.g., Alexa and Siri “sacrifice ‘what we can do’ to ‘what Amazon or Apple allows us to do’.”) This raises questions like, what does a user experience look like where someone can find what they want without Spotlight? And what would need to change about the current tech incentive structure so companies could pursue implementing such a design?
Portable neuroimaging in VR while people are in motion (and playing Beat Saber): Stanford researcher Dr. Cassondra Eng shared an update from her team, demonstrating the ability to measure people’s brain activity while playing Beat Saber in VR. Measuring brain activity while people are moving around as they do in everyday life has historically been difficult (e.g., noise from movement/muscle activity drowning out neural signals and movement displacing electrodes, to name a few challenges), and has limited conducting neuroscience research in natural settings. While I’ll defer a deeper methodological critique for when the full paper is published, it appears Eng’s team is using Functional Near-Infrared Spectroscopy (fNIRS) with a custom algorithm to regress out the head motion and blood flow from the scalp. This seems like a promising development that can help unlock more neuroscience research, both on brain activity during VR use, but also using VR as a kind of “petri dish on your head” to deliver training tasks, audio-visual processing tasks and so on, to help us better understand learning and other cognitive processes.
Community
A shortlist of design research roles in AI/ML, spatial computing or both:
Director of Research & Strategy, Document Cloud | Adobe | Remote & multiple locations
Lead a team of design researchers and strategists who craft experiences across web, mobile, and desktop applications while using AI/ML technologies to enable individual and business productivity.
Human Factors Design Researcher, AI/ML | Apple | Seattle, WA
Help accelerate Apple’s machine learning data annotation efforts by defining user experiences for a suite of software applications and services.
User Experience Researcher | Electronic Arts | Orlando, FL, USA or Vancouver, Canada
Join the team leading UX research for Maxis studio, including The Sims 4 and next-generation of the Sims, Project Rene.
Senior User Experience Researcher | Inflection AI | Palo Alto/Hybrid
Conduct end-to-end research on “personal AI for everyone”, from market discovery to prototype validation.
Mixed Methods UX Researcher, Horizon Worlds | Meta | Los Angeles, CA, Seattle, WA or Burlingame, CA
Help concept, design and ship VR-best experiences for Horizon Worlds, a User Generated Content platform where people can find people, places, and things they care about.
That’s a wrap 🌯 . More human-computer interaction news from Sendfull next week.