Playing Along - Building AI Agents for Co-Creation of Improvised Stories
A novel approach to AI-human collaborative storytelling through improvisation
Abstract
This work focuses on human-agent co-creation of improvised stories, investigating whether Large Language Models can effectively engage in an improvisational practice known as the "Yes! and…" game. We demonstrate how AI systems can participate successfully in improvisational co-creation, moving beyond response generation to collaborative story-telling. We provide a systematic framework for evaluating creative AI outputs in improvisational contexts, combining human evaluation with computational metrics. Our evaluations show that stories co-created with an AI agent are indistinguishable from stories co-created with a human in terms of novelty, value, and surprise. This shows how "hallucinations" – typically considered problematic in AI systems – can serve as creative assets in collaborative storytelling. More generally, our approach presents the "Yes! and…" game as a novel model-system for studying improvised co-creativity in a well-defined and measurable setup.
Examples & Results
Yes! and... Game Example
An example snippet from a "Yes! and..." game showing the collaborative storytelling format where participants build upon each other's contributions.
Creative Ratings Comparison
Comparison of human raters' 1-7 scale ratings over all reviewed games between AI-Human games and Human-Human games, showing the quality assessment of collaborative storytelling.
Novelty Score Analysis
Novelty score comparison between human players and AI players across multiple models. Humans consistently made bigger jumps and scored higher novelty, while AI tended to stay closer to previous turns.
Surprise Score Analysis
Average surprise scores for the three groups: Humans in Human-Human games, Humans in AI-Human games, and AI in AI-Human games. AI contributions were consistently more surprising, and humans playing with AI were also more surprising than humans playing with humans.
Key Contributions
Indistinguishable AI-Human Collaboration
Stories co-created with AI partners received similar ratings to human-human collaborations in external evaluations, and participants in our "Turing test" experiment had difficulty distinguishing between the two conditions. This demonstrates that LLMs can meaningfully engage in structured creative improvisation, contributing to coherent narrative development within the "Yes! and..." game.
Methodological Advances
- Reduced Complexity: The text-based format allows controlled study of core creative dynamics, similar to prior work in movement and visual improvisation.
- Quantifiable Assessment: The experimental setup enables measurement of creative contributions through computational metrics.
- Scalability: The chat-based platform supports systematic large-scale data collection and analysis.
Systematic Framework for Collaborative AI
By integrating improvisational techniques with computational creativity assessment, we establish a systematic framework for developing collaborative AI systems and studying human-AI creative interaction.
Future Work
- Comparing expert improvisers with AI performance.
- Developing systems for dynamic AI contribution control in co-creative processes.
- Implementing real-time evaluation systems for adaptive AI responses.
- Creating platforms for large-scale, unbiased interaction studies.
- Exploring AI's unique creative contributions beyond human-like performance.
Technical Details
Methodology
Our research investigates Large Language Models' ability to engage in the "Yes! and..." improvisational game, a collaborative storytelling practice where participants build upon each other's contributions. We developed AI agents that can participate in this creative process, moving beyond traditional response generation to active co-creation.
Experimental Setup
We conducted systematic evaluations comparing AI-human co-created stories with human-human co-created stories. Our evaluation framework combines human assessment with computational metrics, measuring novelty, value, and surprise across different storytelling scenarios and participant groups.
Results Analysis
Our key finding demonstrates that stories co-created with AI agents are indistinguishable from those co-created with humans across all measured dimensions. This reveals how AI "hallucinations" – typically viewed as problematic – can actually serve as creative assets in collaborative storytelling contexts.
Citation
@inproceedings{vidra2025playing,
title={Playing Along - Building AI Agents for Co-Creation of Improvised Stories},
author={Vidra, Idan Dov and Kimron, Gal and Noy, Lior and Shamir, Ariel},
booktitle={Proceedings of the 16th International Conference on Computational Creativity},
year={2025}
}