What chat GPT can (and cannot) do for UX Research.

The past weekend I finally had a chance to deep-dive into Chat GPT myself. Besides it being able to create backstories for my many D&D characters, I wanted to learn if Chat GPT can aid me in my daily UX Research practices. Here’s what I learned about how AI can and cannot help me in my UX research practices.

How can chat GPT aid me in my UX research?

To test out GPT I decided to look at one of my old research projects. The topic of which was: how student well-being influences student success. When I first did this research, this tool did not exist yet and so all the work for it was done manually. Now with the enhancement of AI, I’m testing different scenarios to see how Chat GPT could help in performing research.

✅ Chat GPT can help you find recent sources on your topic

I asked the AI to provide me with 5 recent articles that prove that student well-being improves student success. Chat GPT replied saying they cannot provide me with articles from the last 5 years. However, they did provide me with 4 out of 5 articles that were from the last 5 years. Because of my query, the AI was capable of finding recent research that would be useful for my research paper. By asking to include the source I could then look up these sources online.

✅ Chat GPT can summarise articles based on your needs

If you want to get a general idea of the sources found, but do not want to spend hours reading all of the articles, then the AI can provide you with brief summaries. This can give you deeper insights into whether or not the article is interesting for your research. While it cannot analyse and tell you if this is a good source for your research, it can provide you with insights into the article. It is then up to you as a UX researcher to determine the quality of the source.

This also works the other way around. Say you have found an interesting article but it’s 100 pages and you really don’t want to spend hours combing through it. You can ask Chat GPT to summarise an article for you. Simply enter the link to the article and ask it for a summary.

Of course, if you want to fully comprehend the source article you should probably read it yourself. But the AI can help you in identifying whether or not the article you found is of interest to you.

✅ Chat GPT can provide suggestions for UX research methods based on your goals

The AI can provide you with suggestions for measuring methods. For this, it needs to know more about the research you’re doing, such as; your target audience, topic, and research goals. This is quite useful as it provides information on not just the method but the sort of data that can be collected as well. When starting a new project, many UX researchers tend to grab the same methods they are familiar with. But by using chat GPT you might consider different methods for your research project.

✅ Chat GPT can give suggestions on how to analyse data

While Chat GPT cannot analyse your research for you, it can provide you with suggestions on how to analyse data. In this case, I was looking for a taxonomy to analyse the data collected from user interviews. The AI provided me with 2 methods for data analysis and an explanation of how they could be used. This is interesting when you are unsure which analysis method you would like to use for your UX Research project. If you have multiple UX researchers working on the same project you could even split the analysis method before triangulating the findings to see if similar findings are found.

✅ Chat GPT can aid in competitors’ research

When the AI understands your topic, goals, and target audience, you can ask it to find related products already out there on the market. This is significantly reducing your time googling for different products. With a list quickly generated, the UX researcher can then decide which products to further look into. This can help you to compare products to your current product, and also to understand unique selling points and possible product gaps.

✅ Chat GPT can aid in setting up a general timeline for your UX research

I was a bit skeptic about asking AI to build me a GANNT chart but I wanted to learn if it was a reasonable request. The results were quite surprising actually. Although chat GPT does not go into detail, it does provide you with a general timeline of tasks. This could be helpful as it shows you the big tasks you need to do to complete this project. However, it is still up to the researcher to define the details in the timeline, such as your methods, protocols, the building of taxonomies etc.

But what is Chat GPT still missing when it comes to research

With current discussions going strong on whether or not AI will replace human jobs, I also wanted to learn what parts of this tool are not helping our research practices or haven’t been developed yet. This indicates that there is still a clear need for human researchers

🚫 Chat GPT cannot (yet) reference in APA

Accurate APA referencing in research papers is very important and so I wanted to find out whether or not the AI can generate a source list based on the APA structure. However, because it doesn’t have a connection to the actual papers (yet) it cannot provide me with proper APA references. Possibly because it does not see the type of article, writers, or DOI numbers. Which is a shame because this part can often take up a lot of time. For now, we will stick to Mendeley for proper sourcing.

Even if you have the full source article yourself, such as the link to it, then chat GPT is still not capable of APA referencing. Indicating that although it can find the articles and read the articles, it cannot see the individual components needed for an accurate APA reference.

🚫 Chat GPT cannot fully build you a research plan

While AI can provide you with a nice overview of a research plan and some simple suggestions, it cannot build a comprehensive research plan for your project. Suggestions for a research question and how to collect data are useful but do not build a detailed plan. If anything, the setup of a research plan is so ingrained in our practices that I do not feel that chat GPT adds much value to it. However, when you are just starting out as a junior UX researcher then AI can provide you with some guidance on where to start.

🚫 Chat GPT cannot build your UX research methods for you

Building UX research methods and their protocols take time. I was curious to understand how Chat GPT could aid in this. Sadly the results here remain a bit superficial. When conducting interviews we first need to know what type of interview structure we want to use. Whether that’s semi-structured, unstructured or fully structured. Then we need to understand the topics we want to interview the users on and formulate questions in a way that help the user open up about their experiences. Chat GPT does not think about these components of creating interview questions, but instead looks at your topic and provided user and gives suggestions for questions.

Looking at the specific questions that the AI provides, we can safely say that chat GPT cannot build me my interview protocols. Asking students to describe their well-being is a complicated question and is charged with cultural bias and experiences. Asking students how they feel about their academic performance will generate questions based on their most recent academic experience. If they have just received a failing grade, their responses will be likely negative and vice versa.

What could be taken from the suggestions of the AI are the topics that are of interest for the interview protocol. But a human UX researcher is needed to build the right questions that will collect the right data from our users.

🚫 Chat GPT cannot analyse your transcripts

Inputting a transcript into chat GPT and asking for a data analysis does not work either. The AI cannot comprehend long amounts of texts, which transcripts often are. So for analysing transcripts we still need a UX Researcher to build a taxonomy and tag the data collected via this.

🚫 Chat GPT cannot analyse human behavior

One of the important human parts of UX research is the ability to analyse human behaviour, such as body language and tone of voice. An understanding of cultural differences in these is also very important when dealing with an international target audience. However, Chat GPT is not yet capable of doing this job for you. Instead, it provides you with some methods for analysing human behavior which can be useful in deciding your method of analysis.

🚫 Chat GPT is sort of biased on solution generation

while asking the AI about useful recommendations from the perspective of product design, chat GPT quickly focuses on providing you with several solutions. These solutions are not based on your research findings and are therefore biased. As we know, AI tools like these pool data from the internet, and so it is highly likely that these are the type of solutions that already exist out there.

What is a good aspect of the AI is that it recognises its own limitations in the UX research department. This highlights again that there is a need for a UX researcher to perform the research, analyse the findings and come up with accurate recommendations for the development of a product. Chat GPT may provide you with suggestions but that does not mean they are the right solutions.

So how exactly does chat GPT change the game for UX research?

As seen in this article, AI can be extremely useful to generate sources and directions to look in. It can provide you with ux research methods that could be interesting for your research, but only if you provide it with the right keywords. Because chat GPT remembers your input, it is important to start out by emphasizing your target audience, research topic, and research goal. With this in mind, the suggestions for sources and methods will be more useful to you.

However, it has been made clear that AI cannot replace a human UX researcher in their job. It is incapable of building plans and protocols, performing the research and more importantly analyse the data collected. When asked what recommendations to look for the AI actually becomes biased in generating product solutions as it does not have your research data to drive insights from.

So while chat GPT can aid in making certain aspects of documentation work easier, we still need a UX researcher for the job of UX Research. I am curious though how AI is going to develop further and what that means for the future of our jobs. Because to me, it would be really interesting if AI can start to recognise user behavior in video material or can see patterns in transcripts of user interviews. I guess we will find out in due time.