User Testing

Description

The functions we tested were the auto translate feature (which includes the relevant setup functions) and the private translation feature (and its relevant setup functions).

For the auto translate feature, we had the user first enter the command to enable auto translation on that channel. After that, the user entered a different command to set the translation language. Finally, the user entered whatever messages they wanted in whatever language they wanted, and it would automatically translate. These translations were sent as regular messages that everyone could see, despite us wanting to use ephemeral messages as our end goal (still figuring it out).

For the private translation feature, we had to user register their default language they wanted their translations in. Then, they entered the /translate command and the message they wanted to translate it. The translations were sent as ephemeral messages.

The code can be found in the https://github.com/UWSocialComputing/Translator.io repository. The commit we used for testing is https://github.com/UWSocialComputing/Translator.io/tree/749c805ddc08228c5613af1ae2e787fc990da79f. We had revamped the database dependency in between this commit and the most recent version, and were experiencing technical issues during the presentation, so we fell back to the older commit which uses a different database dependency, but had the same functionalities.

The other functions, such as the announce command, were not yet implemented and therefore were “wizard-of-oz’d”.

Photos

Findings

We conducted tests with around 4-5 different users. Overall, the testers had a positive experience with the bot, though there were some minor setup issues. These included confusion on how the commands worked, like making a / command and the specific command to call. In addition, when specifying languages to register or choose, our bot required a specific two character string in uppercase for each language. It was a pain to manually check which string to put for each language and took a lot of time and trial + error. The issue of command confusion should be resolved once we implement a /help command that will display more detailed information about the bot and its commands. This will help with understanding how to use the bot. In addition, the language input issue could be fixed with adding another command similar to ‘help’ that displays the input string corresponding to each language that is available to pick. There are also some minor case-handling issues we will need to address, such as adding case-insensitive language abbreviations. During the auto translation we also observed a bug when the users were typing things and it was translating fine, but at the same time when Ahmet typed something it didn’t translate to the correct language, it just kept the same language and didn’t translate it.

For the /translate command, it worked well and there were no issues. We can continue with the current implementation. We also received feedback that could further improve this feature. One piece of feedback is that when the user receives the translation, the bot could also specify the original language of the message as a flag emoji. Another example of feedback we got was that when the user registers for a language that they want, when this is successful we can put more of an explanation of what actually happened and what it means to successfully register a language. As an aside, we recently found out that you can actually get the language each user is using on their actual discord app itself. So, if a user’s discord app is in Spanish, we would be able to find that. We considered just using this instead of having users use the register command and storing that information in a database. We are still undecided about this, but we are leaning towards keeping the status quo for now because users may want to receive translations in a different language than what they are using.

For the auto translation, while the demo went fine and was well-received, we anticipate that we will need to make changes since we want to use ephemeral messages in each user’s preferred language instead of having every user see the translation in just one default language. The problem is that for ephemeral messages, it doesn’t work naturally with auto translation because an ephemeral message can’t be sent without an initial interaction. This issue is still not 100% clear for us, but we are considering having the auto translations be sent as a DM to each individual user as a compromise. We would need to include conversation context so users don’t get lost, so maybe include things such as the server name, channel name, original message, and translated message.

Overall, the main themes of our findings were:

have helper commands for usings to learn how to use the bot
add more detail/context for translations to aid the user
try to make auto-translate feature private, maybe via user DM

After these two features, we will implement the other commands in our code and design spec such as announce, help, getServerLanguages, etc. We also need to finalize our database implementation.