Time

The lesson takes around 30 minutes to complete. There is also a reflection exercise, which is best done jointly with your fellow students.

About the lesson: ‘Algorithms determine your search results’

In today’s digital age, we are dependent on searching for information online via search engines like Google, or we can interact with chatbots like ChatGPT to get answers to our questions. But many people do not realise that the search results presented to us are not a matter of chance. They have been carefully selected and ranked by complex algorithms based on relevance. It is therefore important that university students understand how these algorithms work, so they can critically appraise the search results presented. This lesson explores how algorithms determine your search results and can affect the information you get access to online. Let’s explore this important topic and become more aware of how we can navigate critically within the search results and achieve more reliable and accurate results.

Learning outcomes

When you have completed the lesson, we expect that you:

can explain simply what an algorithm is.
can explain how an algorithm can be biased or ‘overfitted’.
can list examples of factors that affect Google search results.
can explain why you need to critically assess information from search engines like Google and chatbots like ChatGPT.

Source

The lesson was produced by the University of Copenhagen as part of its learning resources for digital literacy, 2023.

Course directors:

Birgitte Munk, special consultant, University Library (KUB)
Erik Engelbrekt Schwägermann, information specialist, University Library (KUB)
Anne Bagger, information specialist, University Library (KUB)
Muhamed Fajkovic, special consultant, University Library (KUB)

__________________________________________________________

What is an algorithm and how does bias arise?

This video explains:

how algorithms can affect human behaviour through automated decision-making and interpretation of huge amounts of data.
why the concepts algorithmic bias and ‘overfitting’ are important.
how algorithms can affect society.

If you experience access denied, try reloading the page.
For English subtitles, please look for the CC icon in the lower right corner of the video and press English.

__________________________________________________________

What happens when you search on Google?

This video explains:

the role algorithms play in the information you find when you search online.
how algorithms influence what you see when you search for information about your assignment

If you experience access denied, try reloading the page.
For English subtitles, please look for the CC icon in the lower right corner of the video and press English.
__________________________________________________________

Google’s algorithms and their functions

Google’s algorithms serve a range of functions, as shown in the chronological overview below:

Read more about algorithms

The oldest algorithm is the PageRank link analysis algorithm. It is also the only transparent one, as it was patented when it first appeared. PageRank determines the popularity of a website based on the number of incoming links, outgoing links and reciprocal linking between sites. The greater the popularity, the higher its position in the search results.

Other algorithms are used to assess the quality of websites, and whether they deserve a high position in the results. This quality assessment is based in part on compliance with Google’s rules and principles for search engine optimisation (see the section on SEO).

In addition to assessing quality and removing misinformation from content pages, which we saw examples of in the video, Google’s algorithms are also used to monitor user data and respond to various situations, e.g. by tracking users’ locations. One example of this is to protect women at abortion clinics in the USA by removing their geolocation information, search history etc. This measure is in response to strict abortion legislation, and concerns about the authorities seizing the data of these women (Ritzau, 2022).

Google search results are thus based on automated decision-making processes. In reality, we know little about how the algorithms work, as they are changed daily and are well-guarded trade secrets. We therefore do not know how the algorithms decide what is misinformation or quality or harmful content, or how often they make mistakes.

__________________________________________________________

Search engine optimisation (SEO)

Search engine optimisation (SEO) is about ‘pleasing’ the algorithms in order to get a good ranking in the results, and thereby attract people to your website without having to pay for expensive advertising. There are two approaches to SEO, one ethical and one non-ethical – also known as ‘white hat SEO’ and ‘black hat SEO’. The differences are explained below:

Read more about SEO

‘White hat SEO’ means that you behave ethically and observe Google’s rules and principles for good search engine optimisation. It may take some time before your website becomes visible, but there is no risk of being punished by being removed from Google’s search results.

‘Black hat SEO’ means you are not interested in a good long-term website ranking, but only in getting a high position in search results with the hope of attracting traffic to your website.

There can be several reasons for this. Here are a few real-life examples:

A well-known businessman received criticism and negative publicity in the news media because of money held in tax havens. This negative coverage was ranking high up in Google’s search results. He therefore hired a marketing company specialising in SEO to make the criticism as invisible as possible and to get websites with positive coverage high up in the results (Jakob Albrecht, 2021).

Another example is the IT-borger.dk domain. This originally belonged to the former IT and Telecoms Agency, and provided information about IT security. A marketing company purchased the domain, and while it continued to provide information about IT security, the real purpose was to help its customers sell IT equipment (Frederik Kulager, 2019).

If Google detects manipulation, fake sites or black hat SEO, the website is punished by being removed from search results and the index, but this can be extremely difficult to detect, even for algorithms. And if caught out, these black hats just create new websites with new domains that will again give them a high ranking in the search results for a time.

__________________________________________________________

Google Scholar

Google rarely displays results from scientific sources, but with Google Scholar, you search among academic publications, including peer-reviewed scientific articles. It therefore makes sense to use Google Scholar rather than Google for scientific information searches for your assignment, but as with Google, there are things you should be aware of (see below):

Read more about Google Scholar

Like Google, it is a web crawler, but it collects and indexes publications from websites with academic and scientific content. As with Google, this is an automated process where algorithms decide what is indexed and presented and how results are ranked.

Like Google, Google Scholar will try to understand what you want from a simple search, and will almost always find something that seems relevant among the first search results. Google Scholar also often finds academic publications in new subject areas that have not yet been added to academic databases, and reports from government agencies and other authoritative sources that an ordinary search on Google will not unearth.

Note that there is generally a lack of clarity about the sources and types of publications that Google Scholar includes, but the algorithms apparently seem to have difficulty distinguishing between fake and genuine scientific publications. For example, articles from various fake journals have been indexed (see the lesson Rubbish or research – beware of predatory journals!).

It is also a mystery how regularly Google Scholar updates its index. For example, a search for various titles in arXiv found that the latest articles in the current month are not yet searchable.

The relevance ranking system is also unclear, but is based in part on the number of citations received by a publication. Unfortunately these are unreliable, as Google Scholar appears to include duplicates and websites with fake articles and citations.

Advanced search options are also lacking, e.g. with Boolean operators and decent filtering, so a fully systematic search is not possible.

Overall, using Google Scholar as the only source of literature is problematic due to a lack of quality control in its coverage and lack of clarity about which sources are indexed.

__________________________________________________________

ChatGPT and other chatbots – Information without sources and fabricated articles

Chatbots are gaining ground and are being increasingly used as an alternative to more traditional search engines. When ChatGPT – the first advanced chatbot – was launched in November 2022, we gained access to a robot that understands and writes human language, and can answer questions and write texts on any conceivable topic.

It offers rapid and concise responses to input such as:
Click on each prompt to see ChatGPT’s reply (ChatGPT Mar 14 Version, 2023):

What are the biggest climate challenges?

ChatGPT - answer icon-1.png ChatGPT replied:

There are several major climate challenges facing the world today. Here are some of the biggest ones:

Global temperature rise: One of the biggest climate challenges is the overall increase in global temperatures, which is causing changes in weather patterns, melting glaciers and sea ice, rising sea levels, and more frequent and severe natural disasters.
Greenhouse gas emissions: Another major challenge is the continued release of greenhouse gases (primarily carbon dioxide) into the atmosphere from human activities such as burning fossil fuels and deforestation. These emissions are causing the Earth’s atmosphere to trap more heat, leading to global warming and other climate impacts.
Loss of biodiversity: Climate change is also having a significant impact on global biodiversity, with many species facing extinction due to changes in temperature, precipitation, and other environmental factors.
Water scarcity: As temperatures rise and precipitation patterns change, water scarcity is becoming an increasingly pressing issue in many regions around the world. This is affecting both human populations and ecosystems, and exacerbating existing water-related challenges such as water pollution and over-extraction.
Food security: Climate change is also impacting food production and distribution, leading to decreased yields and increased food insecurity in many areas. This is particularly challenging for vulnerable populations who rely on agriculture for their livelihoods.

Overall, these climate challenges are interconnected and require urgent and coordinated action to mitigate their impacts and build resilience in the face of continued change.

Please give me 3 scientific articles about water scarcity

ChatGPT - answer icon-1.png ChatGPT replied:

Here are three scientific articles about water scarcity:

“Water scarcity and the impact of improved irrigation management: A comparative analysis across several Mediterranean river basins” by Marta Moneo, et al. This article, published in Agricultural Water Management in 2018, explores the impacts of water scarcity on agriculture in several Mediterranean river basins. The authors use a comparative analysis to examine the effectiveness of different irrigation management strategies in mitigating the impacts of water scarcity on crop yields.
“Assessing global water scarcity and the impact of water conservation measures using a modified water footprint methodology” by Mesfin M. Mekonnen and Arjen Y. Hoekstra. This article, published in Hydrology and Earth System Sciences in 2016, presents a comprehensive assessment of global water scarcity using a modified water footprint methodology. The authors examine the impact of water conservation measures such as reducing food waste and improving irrigation efficiency on water scarcity.
“Water scarcity, food security, and inequality: A longitudinal analysis of global crop production, trade, and consumption” by Kyle Frankel Davis, et al. This article, published in Environmental Research Letters in 2017, analyzes the relationship between water scarcity, food security, and inequality at a global scale. The authors use a longitudinal analysis of crop production, trade, and consumption to identify regions that are particularly vulnerable to water scarcity and food insecurity, and propose policy solutions to address these issues.

While the answers from ChatGPT seem impressive and can help you get a quick overview of a topic, be very aware that you have no idea where the chatbot gets its knowledge from. There are no sources given.

But you need to be even more wary when you ask for references. The three articles referred to in the above example are fabricated. They simply do not exist! When asked about one of the articles, ChatGPT even continues its fabrications like a pathological liar :

ChatGPT - answer icon-1.png ChatGPT replied:
The article “Water scarcity and the impact of improved irrigation management: A comparative analysis across several Mediterranean river basins” was published in the journal Agricultural Water Management, Volume 97, Issue 4, Pages 503-512, in March 2010. The authors of the article are Andreas Th. Bruggeman, Raffaele Cortignani, Gianluca Dono, Andrea Daccache, and Ariel Dinar.

Here is the precise citation for the article:

Bruggeman, A. T., Cortignani, R., Dono, G., Daccache, A., & Dinar, A. (2010). Water scarcity and the impact of improved irrigation management: A comparative analysis across several Mediterranean river basins. Agricultural Water Management, 97(4), 503-512. https://doi.org/10.1016/j.agwat.2009.10.017

I can’t find this article. Are you sure it is real?

ChatGPT - answer icon-1.png ChatGPT replied:

I apologize for any confusion I may have caused. After double-checking, I found that there was an error in my previous response. I apologize for this mistake.

The article “Water scarcity and the impact of improved irrigation management: A comparative analysis across several Mediterranean river basins” was published in the journal Agricultural Water Management, Volume 97, Issue 4, Pages 503-512, in March 2010. The authors of the article are Andreas Th. Bruggeman, Raffaele Cortignani, Gianluca Dono, Andrea Daccache, and Ariel Dinar.

I apologize for the inconvenience and confusion caused by my previous response.

How does ChatGPT work?

ChatGPT is a ‘Large Language Model’ – a generative AI language model, developed and trained to generate text responses that mimic human chat conversations. The model has been trained using very large volumes of text from every corner of the internet – chat conversations, digitised books, research articles, news articles, social media content etc. Once the developers of an AI language model are satisfied with its ability to generate good human-like responses, it no longer has access to internet texts, and thus does not look for answers in the same way as a traditional search engine.

Instead, the model uses a complex system of algorithms to compose words and sentences into meaningful answers, based on all the text it has previously digested. This is why ChatGPT makes up the scientific articles shown above. It connects words to create something similar to a real title, author list, journal, year and DOI link – but in most cases it misses the mark, at least to date.

Read more about how generative AI language models are developed on KUnet > The technology behind ChatGPT.

The technology behind and applications of AI language models such as ChatGPT, the new Microsoft Bing and Google’s Bard, are rapidly evolving. The language models are gradually being integrated into other solutions, such that search engines like Google formulate their own responses, rather than just presenting links to relevant sources. You can see an example of this in the ‘What happens when you search on Google?’ video above.

You should therefore be aware that the search results and answers you receive from search engines and chatbots are controlled by algorithms designed for purposes other than transparent scientific information searches.

__________________________________________________________

Always check the sources – or use recognised academic databases

Chatbots, Google, Google Scholar and other search engines that broadly search the internet are not reliable tools for academic information searches. It is therefore important that you always check the information and articles you find via chatbots and search engines. You can learn more about this in the lessons: ’Basic source criticism’ and ‘New methods for digital source criticism’.

Alternatively, you can use recognised academic databases, where journals are indexed under human control, and you can conduct a systematic search that is free of algorithms driven by commercial interests.

University students in Denmark have free access to search in academic databases via the Royal Library’s database at soeg.kb.dk. Your teachers can also help you find good academic databases for your subject area.

__________________________________________________________

Reflection exercise

Search for a topic, such as “climate solutions”, on Google and reflect on these questions:

Are the best sources shown at the top of Google’s list of results? If not, what kind of results are shown?
What does Google’s ranking of results reflect (quality, reliability, importance to you, popularity)?
What are the advantages/disadvantages of automated search results?

Perform the same search on Google Scholar and reflect on these questions:

What are the similarities and differences between the search results from Google and Google Scholar?
In which contexts is Google Scholar suitable for academic literature searches?

You may also want to ask a chatbot like ChatGPT: “What are the best climate solutions?” Reflect on the following:

What value does the chatbot’s answer have in relation to scientific information searches?
Would you ever use text from a chatbot in an academic assignment? How/why/why not?

____________________________________________________________

If you want to know more

Haider, J. og O. Sundin. 2019. Invisible Search and Online Search Engines. Routledge. https://doi.org/10.4324/9780429448546.
Halevi, G., Henk M. & J. Bar-Ilan. 2017. “Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature”. Journal of Informetrics 11 (3): 823–34. https://doi.org/10.1016/j.joi.2017.06.005.
Morris, W. 2022. “A Primer on SEO Techniques: The Good, The Bad, and The Ugly”. Elegant Themes Blog https://www.elegantthemes.com/blog/wordpress/seo-techniques.

__________________________________________________________