The Effect of Question Formats in Dialogue with Generative AI: A Comparative Analysis of Open Questions and Prompt Engineering

Keisuke Sato

doi:10.51094/jxiv.853

##article.authors##

Keisuke Sato Natural Science, National Institute of Technology, Ibaraki College https://orcid.org/0000-0002-0469-8077 https://researchmap.jp/skeisuke

DOI:

https://doi.org/10.51094/jxiv.853

Keywords:

Generative AI, Question formats, Open-ended questions, Prompt engineering, Human-AI interaction

Abstract

As dialogue with generative AI enters a new phase, this study compares and analyzes the effects of question formats in AI interactions, specifically comparing open-ended questions with prompt engineering (structured questions). We conducted experiments using eight AI models in four areas: the future of education, improving communication within companies, family meal planning, and new movies, and analyzed them using 20 evaluation metrics.
The results showed that the question format had a multifaceted and complex effect on the AI response. Open questions showed an advantage in terms of creativity, diversity, and thought promotion, while prompt engineering formats were effective in terms of concreteness and naturalness of dialogue. In addition, the effectiveness of the question format was highly dependent on the theme and AI model used.
These findings suggest the importance of strategic selection of question formats in designing interactions with AI. Choosing the appropriate question format according to the purpose, theme, and AI model used can lead to more effective and creative interactions.
This research provides new guidelines for effective interaction with AI and emphasizes the importance of improving AI literacy.

Conflicts of Interest Disclosure

The author declares no competing interests.

Downloads *Displays the aggregated results up to the previous day.

Download data is not yet available.

Author Biography

Keisuke Sato, Natural Science, National Institute of Technology, Ibaraki College

April 2015 - Present
Associate Professor, Ibaraki National College of Technology

April 2010 - March 2015
Lecturer, Ibaraki National College of Technology

April 2008 - March 2010
Specially Appointed Researcher, Institute for Solid State Physics, The University of Tokyo

April 2002 - March 2008
Researcher, Fujitsu Laboratories Ltd.

References

Eapen, T., Finkenstadt, D.J., Folk, J., Venkataswamy, L.: How generative ai can augment human creativity. Harvard Business Review 101(4) (2023) 25

Rayan, J., Kanetkar, D., Gong, Y., Yang, Y., Palani, S., Xia, H., Dow, S.P.: Exploring the potential for generative ai-based conversational cues for real-time collaborative ideation. In: Proceedings of the 16th Conference on Creativity & Cognition, pp. 117–131 (2024)

Yang, D.: Human-ai interaction in the age of large language models. In: Proceedings of the AAAI Symposium Series, vol. 3, pp. 66–67 (2024)

Bozkurt, A.: Tell me your prompts and I will make them true: The alchemy of prompt engineering and generative AI. International Council for Open and Distance Education Oslo, Norway (2024). https://doi.org/10.55982/openpraxis.16.2.661

Korzy´nski, P., Mazurek, G., Krzypkowska, P., Kurasi´nski, A.: Artificial intelligence prompt engineering as a new digital competence: Analysis of generative ai technologies such as chatgpt. Entrepreneurial Business and Economics Review (2023) https://doi.org/10.15678/eber.2023.110302

White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023). https://arxiv.org/abs/2302.11382

White, J., Hays, S., Fu, Q., Spencer-Smith, J., Schmidt, D.C.: ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design (2023). https://arxiv.org/abs/2303.07839

Wit, A.: ‘when i’m retired...’: Using topic modeling to analyze open-ended survey questions in the a broader mind longitudinal survey. (2021). https://doi.org/10.31235/osf.io/ng4m2 . https://api.semanticscholar.org/CorpusID:240821164

Haq, I.U., Pifarr´e, M., Fraca, E.: Novelty evaluation using sentence embedding models in open-ended cocreative problem-solving. International Journal of Artificial Intelligence in Education (2024) https://doi.org/10.1007/s40593-024-00392-3

Ecoffet, A., Clune, J., Lehman, J.: Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity (2020). https://arxiv.org/abs/2006.07495

Chen, B., Zhang, Z., Langren´e, N., Zhu, S.: Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735 (2023)

Boussioux, L., Lane, J.N., Zhang, M., Jacimovic, V., Lakhani, K.R.: The crowdless future? generative ai and creative problem solving. Working Paper 24-005, Harvard Business School Technology & Operations Mgt. Unit (July 2024). https://doi.org/10.2139/ssrn.4533642 . https://ssrn.com/abstract=4533642

Doshi, A.R., Hauser, O.P.: Generative ai enhances individual creativity but reduces the collective diversity of novel content. Science Advances 10(28), 5290 (2024) https://doi.org/10.1126/sciadv.adn5290 https://www.science.org/doi/pdf/10.1126/sciadv.adn5290

Perifanis, N.-A., Kitsios, F.: Investigating the influence of artificial intelligence on business value in the digital era of strategy: A literature review. Information 14(2) (2023) https://doi.org/10.3390/info14020085

Markowitz, D.M., Boyd, R.L., Blackburn, K.: From silicon to solutions: Ai’s impending impact on research and discovery. Frontiers in Social Psychology 2 (2024) https://doi.org/10.3389/frsps.2024.1392128

Anthropic: Claude prompting guide.md. Claude AI assistant. Unpublished document, Claude Example Project, as of August 18, 2024 (2024)

Shimabucoro, L., Ruder, S., Kreutzer, J., Fadaee, M., Hooker, S.: LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives (2024). https://arxiv.org/abs/2407.01490

Oppenlaender, J., Linder, R., Silvennoinen, J.: Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering (2024). https://arxiv.org/abs/2303.13534

Marrone, R., Taddeo, V., Hill, G.: Creativity and artificial intelligence—a student perspective. Journal of Intelligence 10(3), 65 (2022) https://doi.org/10.3390/jintelligence10030065