Background: Adolescents and young adults are disproportionately affected by HIV, suggesting that HIV prevention methods such as pre-exposure prophylaxis (PrEP) should focus on this group as a priority. As digital natives, youth likely turn to internet resources regarding health topics they may not feel comfortable discussing with their medical providers. To optimize informed decision-making by adolescents and young adults most impacted by HIV, the information from internet searches should be educational, accurate, and readable.
Objective: The aims of this study were to compare the accuracy of web-based PrEP information found using web search engines and virtual assistants, and to assess the readability of the resulting information.
Methods: Adolescent HIV prevention clinical experts developed a list of 23 prevention-related questions that were posed to search engines (Ask.com, Bing, Google, and Yahoo) and virtual assistants (Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri). The first three results from search engines and virtual assistant web references, as well as virtual assistant verbal responses, were recorded and coded using a six-tier scale to assess the quality of information produced. The results were also entered in a web-based tool determining readability using the Flesch-Kincaid Grade Level scale.
Results: Google web search engine and Google Assistant more frequently produced PrEP information of higher quality than the other search engines and virtual assistants with scores ranging from 3.4 to 3.7 and 2.8 to 3.3, respectively. Additionally, the resulting information generally was presented in language at a seventh and 10th grade reading level according to the Flesch-Kincaid Grade Level scale.
Conclusions: Adolescents and young adults are large consumers of technology and may experience discomfort discussing their sexual health with providers. It is important that efforts are made to ensure the information they receive about HIV prevention methods, and PrEP in particular, is comprehensive, comprehensible, and widely available.
The HIV epidemic is a major health concern, with the Centers for Disease Control and Prevention (CDC) reporting 37,698 new HIV diagnoses in the United States in 2018 [, ]. While individuals aged 25-44 years had the highest incidence of HIV, youth aged 13-24 years comprised more than half of all new diagnoses [ ].
Pre-exposure prophylaxis (PrEP) is an effective daily prevention method and should be considered for use by HIV-negative populations at higher risk of HIV . When taken as prescribed, PrEP can offer almost complete protection against HIV; however, previous studies have shown that PrEP uptake is lower in higher risk populations [ , ]. Barriers to PrEP uptake include limited awareness and limited access as a result of PrEP marketing and additional socioeconomic challenges, respectively [ ].
Moreover, adolescents and young adults are major consumers of technology and often rely on the internet for health information [, - ]. Since adolescents and young adults may be consulting the internet instead of health care providers for HIV prevention information, it is critical that health information on the web is accurate and readable. To the authors’ knowledge, there is limited research assessing the quality of PrEP information available on the web. Therefore, the objectives of this study are to determine if accurate and readable PrEP information can be found using web search engines and virtual assistants. Additionally, this study aimed to assess how PrEP information found via both methods compared among and between search engines and virtual assistants.
The institutional review board at Children’s Hospital of Philadelphia determined that approval was not necessary for this study as it did not contain human subjects.
Developing a Framework for Evaluating PrEP Internet Content
Common PrEP information themes were determined for future searches within the United States using English only: PrEP basics, PrEP access, and PrEP use (). A total of 50 questions were compiled through an iterative process with an expert panel of adolescent medicine researchers and physicians who work with adolescents and young adults at high risk of contracting HIV as well as youth living with HIV. Items were narrowed down to 23 questions designed to address concerns adolescents and young adults may have about general use, privacy, and confidentiality ( ).
Pre-exposure prophylaxis (PrEP) basics
- The questions in this category aim to find general information about PrEP for HIV prevention to introduce and educate potential users about this intervention method.
- The questions in this category aim to find information about ways users can access PrEP in addition to information about privacy concerns.
- The questions in this category aim to find information regarding the proper use of PrEP so that potential users can receive the maximum benefits.
|What is PrEPa?||Basics|
|What kinds of PrEP are available?||Basics|
|What is Truvada?||Basics|
|What is Descovy?||Basics|
|What is the difference between Truvada and Descovy?||Basics|
|Does PrEP protect against other sexually transmitted infections/sexually transmitted diseases?b||Basics|
|Should I take PrEP?||Basics|
|What are the side effects of taking PrEP?||Basics, use|
|Will PrEP make me feel sick?||Basics|
|How does PrEP work?||Basics, use|
|What is the difference between PEPc and PrEP?||Basics|
|How effective is PrEP?||Basics, use|
|Where can I find PrEP?||Access|
|How can I get PrEP?||Access|
|Do I need insurance to get PrEP?||Access|
|Do I need my parents’ permission to get PrEP?||Access|
|Will my parents know that I’m using PrEP if I’m on their insurance?||Access|
|How much does PrEP cost?||Access|
|How do I take PrEP?||Use|
|When can I stop taking PrEP?||Use|
|How long does it take for PrEP to start working?||Use|
|What happens if I miss a dose of PrEP?||Use, basics|
|How do I refill my PrEP prescription?||Use, access|
aPrEP: pre-exposure prophylaxis.
bThis question was asked using both “sexually transmitted infections” and “sexually transmitted diseases” due to the recent change in terminology.
cPEP: postexposure prophylaxis.
Data Collection: Web Search Engines and Virtual Assistants
Questions were entered into search engines—Ask.com, Bing, Google, and Yahoo—using a private Mozilla Firefox browser and verbally posed to virtual assistants—Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri—using new user accounts apart from Apple Siri where a pre-existing account with cleared browsing history was used. Questions were asked to search engines and virtual assistants on separate days in March 2021 with a maximum of 10 times for virtual assistants to generate a response. Excluding advertised content, information from the first three web search engine results and virtual assistant nonverbal results were transcribed as listed on the website and recorded in corresponding Excel (Microsoft Corporation) sheets for future analysis. Verbal responses from each virtual assistant were recorded, transcribed, and saved in Excel.
Data Analyses: Scale Development, Rating System for Information Accuracy, and Readability Assessment
A six-tier scale was created and reviewed by an adolescent medicine physician and a researcher to code example content (). The examples were used as content validation to ensure that results were coded using the same criteria. Results received two scores from two different coders who assigned final scores after reaching a consensus. Final scores were then used to calculate an average quality score for each web-based resource in each category. The data collected were also entered into a web-based literacy measurement tool [ ] to assess readability using the Flesch-Kincaid Grade Level Scale [ ].
|Rating number||Meaning||Example content (question: what is PrEPa?)|
|0||The resulting answer did not include any information about PrEP.||“HIV (human immunodeficiency virus) is a virus that attacks the body’s immune system. There is currently no effective cure. Once people get HIV, they have it for life” |
|1||The resulting answer included information about PrEP that was completely inaccurate.||“PrEP is a medication taken only right before sex in order to prevent HIV. While taking PrEP, individuals do not need to use condoms” |
|2||The resulting information about PrEP was partially correct but contained some errors/misleading information. The source providing information was unreliable.||“PrEP is a pill taken before sex, so it is pre-exposure. Prophylaxis means to prevent infection. So you can use PrEP to greatly reduce the risk of becoming HIV positive” |
|3||The resulting information was accurate but lacked many important details.||“Pre-exposure Prophylaxis, also known as PrEP, is when people take medicine to lower their risk of getting HIV” |
|4||The resulting information was detailed, accurate, and from a reliable source. However, the information included was outdated.||“PrEP is an experimental approach to HIV prevention and consists of antiretroviral drugs to be taken before potential HIV exposure in order to reduce the risk of HIV infection and continued during periods of risk” |
|5||The resulting information was detailed, accurate, and from a reliable source. Additionally, the result included the latest information available.||“Pre-exposure prophylaxis (or PrEP) is a way for people who do not have HIV but who are at very high risk of getting HIV to prevent HIV infection by taking a pill every day. The pill (brand name Truvada) contains two medicines (tenofovir and emtricitabine) that are used in combination with other medicines to treat HIV. When someone is exposed to HIV through sex or injection drug use, these medicines can work to keep the virus from establishing a permanent infection. When taken daily, PrEP is highly effective for preventing HIV. Studies have shown that PrEP reduces the risk of getting HIV from sex by about 99% when taken daily. Among people who inject drugs, PrEP reduces the risk of getting HIV by at least 74% when taken daily. PrEP is much less effective if it is not taken consistently. As PrEP only protects against HIV, condoms are important for the protection against other STDs. Condoms are also an important prevention strategy if PrEP is not taken consistently.” |
aPrEP: pre-exposure prophylaxis.
Web-Based PrEP Information Quality
From all search engines and virtual assistants, 422 results were compiled and assigned a score from the PrEP Information Quality Scale (). The results show that Google search engine and Google Assistant more frequently provided higher quality information than resources from other companies, and that search engines provided higher quality information than virtual assistants ( ). Google search engine provided more comprehensive results directing potential users to websites with information on the ideal candidates, proper use, and access. However, Ask.com often provided the lowest quality information directing users to websites containing no information about PrEP. Bing and Yahoo provided information quality between Google and Ask.com.
|Resource||PrEPb basics||PrEP access||PrEP use|
aPrEP information score meanings: 0=irrelevant, 1=inaccurate, 2=partially correct/misleading, 3=accurate but missing important details, 4=accurate but outdated, 5=accurate and current.
bPrEP: pre-exposure prophylaxis.
The following responses were given to the question “How much does PrEP cost?”
...A month’s supply of Truvada is nearly $2,000 without insurance. Most private health insurance companies, Medicare, and Medicaid will cover the cost...
[WebMD via Google]
Launching a startup...
Within virtual assistants, Google Assistant typically provided more informative results when compared to the other virtual assistants selecting relevant text from websites to respond to posed questions. Amazon Alexa generally provided the lowest quality information with irrelevant responses including “Sorry, I don’t know that one.” Microsoft Cortana and Apple Siri provided information quality between Google Assistant and Amazon Alexa.
The following responses were given to the question “What is the difference between Truvada and Descovy?”
There are now two medications approved by the U.S Food and Drug Administration (FDA): Truvada and Descovy...which medication might be right for you, take a look at the similarities and differences between the two medications below...
[SFAF.org via Google Assistant]
Pharmaceutical product. Star Trek Discovery...
Web-Based PrEP Information Readability
The readability assessment using the Flesch-Kincaid Grade Level scale revealed that information from web search engines produced by questions in all three categories requires a literacy level that aligns with that of an eighth grade, ninth grade, or 10th grade student, while virtual assistant results require literacy levels ranging from a second grade to eighth grade student in the United States (). The required literacy level of information produced by search engines and virtual assistants appears to correlate with the quality of information produced by each, as the search engines more frequently provided information of higher quality than the virtual assistants.
|Resource||PrEPb basics||PrEP access||PrEP use|
aPrEP information readability score formula: grade level = 0.37 (words/sentence) + 5.84 (syllables/word) – 15.59.
bPrEP: pre-exposure prophylaxis.
Google search engine and Google Assistant produced higher quality PrEP information more frequently than the other web-based resources used. Authors found resulting answers to PrEP questions that conflicted with each other within a single search engine/virtual assistant, which may lead to incorrect use of PrEP and ultimately reduced effectiveness. Additionally, the resulting information generally was presented in language between a seventh and 10th grade reading level, therefore, often exceeding the average reading level of adolescents and young adults in the United States .
Comparison With Prior Work
To the authors’ knowledge, there are no other studies that have specifically evaluated the quality of PrEP information for HIV prevention produced by virtual assistants or the readability of health information found on the web. However, a New Zealand study titled “In Bed with Siri and Google Assistant: A Comparison of Sexual Health Advice” used Google search engine, Google Assistant, and Apple Siri to assess the quality of sexual health advice received from each source. The results of this study are consistent with researchers’ findings that Google search engine had the best responses, followed by Google Assistant then Apple Siri, when asked questions about sex and sexual health .
The study limitations include the variability of results produced by search engine and virtual assistant algorithms. To account for the possibility of different top results for the same questions, searches were performed on a single day for all search engines and on a separate day within the same week for all virtual assistants. Future studies may consider conducting a time-based analysis where searches are completed at set time intervals over 1 year or more to compare top results for the same questions over time. Another limitation is the condensed list of questions. Future studies should include youth perspectives on what questions adolescents and young adults would ask in addition to provider opinions. Additionally, the study was conducted in English, which may exclude individuals whose primary language is not English. Furthermore, at the time of this study, injectable PrEP had not yet been approved and are not included in the search results.
Adolescents and young adults may turn to technology due to discomfort discussing their sexual health with providers, yet we found that there is much room for improvement in the quality and readability of educational information on PrEP through web-based resources.
This research was directly supported by grants from the National Institute of Mental Health P30MH097488 (principal investigator: Dwight Evans) and K23MH119976-01A1 (principal investigator: SW) and the Leadership Education in Adolescent and Young Adult Health program.
Conflicts of Interest
- HIV surveillance report, 2018 (updated). Centers for Disease Control and Prevention. 2020. URL: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-report-2018-updated-vol-31.pdf [accessed 2023-06-27]
- HIV surveillance report. Centers for Disease Control and Prevention. 2020. URL: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-report-2018-preliminary-vol-30-infographic.pdf [accessed 2023-06-27]
- Pre-exposure prophylaxis (PrEP). HIVinfo. 2020. URL: https://hivinfo.nih.gov/understanding-hiv/fact-sheets/pre-exposure-prophylaxis-prep [accessed 2021-01-14]
- PrEP effectiveness. Centers for Disease Control and Prevention. 2020. URL: https://www.cdc.gov/hiv/basics/prep/prep-effectiveness.html [accessed 2021-01-19]
- Wood SM, Lee S, Barg FK, Castillo M, Dowshen N. Young transgender women's attitudes toward HIV pre-exposure prophylaxis. J Adolesc Health 2017 May;60(5):549-555 [https://europepmc.org/abstract/MED/28132744] [CrossRef] [Medline]
- Hargittai E, Hinnant A. Digital inequality: differences in young adults' use of the internet. Commun Res 2008 Aug 04;35(5):602-621 [CrossRef]
- Escoffery C, Miner KR, Adame DD, Butler S, McCormick L, Mendell E. Internet use for health information among college students. J Am Coll Health 2005 Aug 06;53(4):183-188 [CrossRef] [Medline]
- Anderson M, Jiang J. Teens, social media and technology 2018. Pew Research Center. 2018. URL: https://www.pewresearch.org/internet/wp-content/uploads/sites/9/2018/05/PI_2018.05.31_TeensTech_FINAL.pdf [accessed 2021-02-11]
- Kydala Publishing Inc. StoryToolz. URL: https://storytoolz.com [accessed 2021-03-17]
- Kincaid JP, Fishbourne Jr. RP, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) for Navy enlisted personnel. Institute for Simulation and Training 1975 Jan 01;56:i-39 [CrossRef]
- About HIV. Centers for Disease Control and Prevention. 2020. URL: https://tinyurl.com/3c7nmxsr [accessed 2021-02-01]
- PrEP myths. Houston Health Department. 2019. URL: https://houstoniamlife.com/prep/prep-myths/ [accessed 2021-02-01]
- What is PrEP? HIV i-Base. 2019. URL: https://i-base.info/guides/prep/what-is-prep [accessed 2021-02-01]
- What is PrEP? National Women’s Health Network. 2018. URL: https://nwhn.org/what-is-prep/ [accessed 2021-02-01]
- Naswa S, Marfatia YS. Pre-exposure prophylaxis of HIV. Indian J Sex Transm Dis AIDS 2011 Jan;32(1):1-8 [CrossRef] [Medline]
- The basics - PrEP. Prepfacts. 2020. URL: https://www.cdc.gov/hiv/risk/prep/index.html#print [accessed 2021-04-06]
- Marchand L. What is readability and why should content editors care about it? Center for Plain Language. 2017. URL: https://tinyurl.com/3vykwjze [accessed 2021-04-02]
- Wilson N, MacDonald EJ, Mansoor OD, Morgan J. In bed with Siri and Google Assistant: a comparison of sexual health advice. BMJ 2017 Dec 13;359:j5635-j5642 [CrossRef] [Medline]
|CDC: Centers for Disease Control and Prevention|
|PrEP: pre-exposure prophylaxis|
Edited by S Badawy; submitted 09.09.22; peer-reviewed by U Belkind; comments to author 04.03.23; revised version received 31.03.23; accepted 01.04.23; published 18.07.23Copyright
©Kaja Darien, Susan Lee, Kayla Knowles, Sarah Wood, Miriam D Langer, Nellie Lazar, Nadia Dowshen. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 18.07.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.