Article in partnership with Day Translations. Being an Italian native speaker I was curious to test ChatGPT’s translating functions and before I started writing this article my expectations were low to say the least, as I imagined the computation results to be a real dog’s dinner. You will find my verdict in the final section of this article.

ChatGPT from OpenAI

Here’s a quick recap of the story of ChatGPT so far. The company behind the app says on its website’s mission statement that it has developed its artificial intelligence solution to benefit all humanity. The product announcement for ChatGPT dates 30^th November 2022 and the statement says that the app would be available for free initially to collect user feedback and to make improvements and changes as necessary.

To create an account you go to their dedicated web page where you will see a prompt to register. You are presented with options that include creating an account with your email, your Google profile, your Microsoft profile or your Apple account. You are then asked to verify your phone number by text. After login, you will see a screen displaying a message that you are accessing a free research preview. At the time of writing, there is no mention of how long the free trial is likely to last. There is also another display message warning users that there may be the occasional error or misleading information in the content being rendered. For legal reasons, OpenAI concludes the warning message by stating that the information provided by the app ChatGPT “is not intended to give advice”. Users are also warned not to share sensitive information in their conversations within ChatGPT. Users can rate how useful or not the response from the app was, too.

A final note: the information contained within ChatGPT to provide answers to users’ questions is limited to up to the year 2021 with limited knowledge of events and facts after that year.

ChatGPT in Italian

ChatGPT supports multiple languages including Italian. Because ChatGPT is designed to simulate a real conversation among humans, it is based on NLP or natural language processing.

Automated text from ChatGPT aims at mimicking human conversations in a way that appears seamless and genuine, i.e., not generated by a computer. One of the automated functions of ChatGPT is multiple language translation. Looking at reviews from end users in languages other than English, the general perception is that the results being rendered are less precise and are likely to contain more errors than their English counterpart.

Example 1: Regional Italian Soup Recipe

My very first question to ChatGPT was to translate into English a recipe for an Italian soup popular in my home town. The app produced the recipe with ingredients and instructions within a second. The content of the recipe read well without errors. I then asked ChatGPT to translate a few sentences from the recipe’s instructions back into Italian and I rejoyced at seeing some mistakes, mainly a poorly translated adjective for cooked onion that was picked for its similar spelling to its English equivalent but which wouldn’t feature in a recipe (“traslucida” instead of “semitrasparente”). The sentences also sounded quite alien because they followed an English structure and did not represent what “natural language” would sound like between Italian speakers. I wanted ChatGPT to explain what “traslucida” meant and I asked the question in Italian. I received a reply in Italian within a second explaining that it meant not quite transparent but cooked to the point when the consistency becomes soft. This response made me feel like I was being schooled in my own native language, with the finger-wagging app assuming an air of superiority as it talked down to me! I then asked about one of the ingredients and what size to chop it for the soup, with the answer being that it depends on personal preferences, giving a range of 1-2cm. I thought that the “it depends” reply was a lazy way to just blurt out the information, which also had duplication between the opening sentence and the last sentence using some filler text with minor changes in between. The rest of the explanation sounded like a literal translation from English to Italian.

I also asked about replacing one ingredient with another, namely canned beans instead of dried beans, and the app mentioned in the reply “una consistenza più consistente” which sounds awful in Italian (roughly translated as “a more solid consistency” or, “a thicker thickness”).

Example 2: Hayfever

At the time of writing it’s hayfever season so I was curious to see how ChatGPT would describe the symptoms of hayfever in Italian. I asked the question in Italian and the reply came within a second, most likely collating information from health websites because the language sounded more like commonly spoken Italian. I thought the explanation was clear and to the point.

Example 3: Who Won Eurovision This Year

I wanted to test the database and I asked in Italian about the most recent Eurovision Song Contest winner and the answer, in Italian, said that the app does not have access to real time information as the stored data was last updated in September 2021. The app in its reply first apologised for not having access to that information and concluded with a suggestion to check trustworthy and reputable sources such as the Eurovision Song Contest’s official website.

My Verdict

Considering I had never used ChatGPT in English before I was generally curious about how text in languages other than English was being rendered.

Like with anything else involving data, the quality of what you put in will be reflected in the quality of what you get at the end. It is important to write good prompts and questions, refine these questions as you go along asking for clarification and further information. It seems that creating lists is one of ChatGPT’s strengths, which can be useful when you are stuck for inspiration. However, the quality of the translated content left much to be desired and the accuracy of the translation depends on the source material that the app can access. In other words, topics that have been covered extensively are likely to produce better results than those sourced from technical documents and about specialised subjects.

As I mentioned in other articles before, I strongly believe that translations must be allocated to qualified, professional human translators. For example, if a company wants to save money on copy writing in different languages by deploying only automated tools, the savings would soon be overtaken by the costs related to the reputational damage of publishing misleading or incorrect statements. It is important for companies that have a global strategy to plan for the long term instead of looking at short term budget cuts which can have a lasting impact on the customers’ trust in a brand, product or service.

On a final note: no, ChatGPT didn’t write this article.