This paper is in the following e-collection/theme issue:

Pediatrics (121) Infodemiology and Infoveillance (1269) Quality/Credibility of eHealth Information and Trust Issues (485) Information Retrieval (271) HIV/AIDS/STI Prevention and Care (816) Public (e)Health, Digital Epidemiology and Public Health Informatics (636) Chatbots and Conversational Agents (458)

Published on in Vol 6 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/41806, first published .
Health Information From Web Search Engines and Virtual Assistants About Pre-Exposure Prophylaxis for HIV Prevention in Adolescents and Young Adults: Content Analysis

Health Information From Web Search Engines and Virtual Assistants About Pre-Exposure Prophylaxis for HIV Prevention in Adolescents and Young Adults: Content Analysis

Health Information From Web Search Engines and Virtual Assistants About Pre-Exposure Prophylaxis for HIV Prevention in Adolescents and Young Adults: Content Analysis

Authors of this article:

Kaja Darien1 Author Orcid Image ;   Susan Lee1, 2 Author Orcid Image ;   Kayla Knowles1 Author Orcid Image ;   Sarah Wood1, 2, 3 Author Orcid Image ;   Miriam D Langer2 Author Orcid Image ;   Nellie Lazar3 Author Orcid Image ;   Nadia Dowshen1, 2, 3 Author Orcid Image
Article Authors Cited by (1) Tweetations (4) Metrics

    Original Paper

    1PolicyLab, Children's Hospital of Philadelphia, Roberts Center for Pediatric Research, Philadelphia, PA, United States

    2Craig-Dalsimer Division of Adolescent Medicine, Children's Hospital of Philadelphia, Philadelphia, PA, United States

    3Department of Pediatrics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States

    *all authors contributed equally

    Corresponding Author:

    Kaja Darien, BA

    PolicyLab

    Children's Hospital of Philadelphia

    Roberts Center for Pediatric Research

    2716 South Street

    10th Floor

    Philadelphia, PA, 19146

    United States

    Phone: 1 267 426 9523

    Email: darienk@chop.edu


    Background: Adolescents and young adults are disproportionately affected by HIV, suggesting that HIV prevention methods such as pre-exposure prophylaxis (PrEP) should focus on this group as a priority. As digital natives, youth likely turn to internet resources regarding health topics they may not feel comfortable discussing with their medical providers. To optimize informed decision-making by adolescents and young adults most impacted by HIV, the information from internet searches should be educational, accurate, and readable.

    Objective: The aims of this study were to compare the accuracy of web-based PrEP information found using web search engines and virtual assistants, and to assess the readability of the resulting information.

    Methods: Adolescent HIV prevention clinical experts developed a list of 23 prevention-related questions that were posed to search engines (Ask.com, Bing, Google, and Yahoo) and virtual assistants (Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri). The first three results from search engines and virtual assistant web references, as well as virtual assistant verbal responses, were recorded and coded using a six-tier scale to assess the quality of information produced. The results were also entered in a web-based tool determining readability using the Flesch-Kincaid Grade Level scale.

    Results: Google web search engine and Google Assistant more frequently produced PrEP information of higher quality than the other search engines and virtual assistants with scores ranging from 3.4 to 3.7 and 2.8 to 3.3, respectively. Additionally, the resulting information generally was presented in language at a seventh and 10th grade reading level according to the Flesch-Kincaid Grade Level scale.

    Conclusions: Adolescents and young adults are large consumers of technology and may experience discomfort discussing their sexual health with providers. It is important that efforts are made to ensure the information they receive about HIV prevention methods, and PrEP in particular, is comprehensive, comprehensible, and widely available.

    JMIR Pediatr Parent 2023;6:e41806

    doi:10.2196/41806

    Keywords

    pre-exposure prophylaxis; PrEP; prophylaxis; internet use; search engine; adolescent; youth; pediatric; adolescence; young adult; readability; human immunodeficiency virus; HIV; virtual assistant; health information; information quality; accuracy; credibility; patient education; comprehension; comprehensible; web-based; online information; sexual health; reading level 


    The HIV epidemic is a major health concern, with the Centers for Disease Control and Prevention (CDC) reporting 37,698 new HIV diagnoses in the United States in 2018 [1,2]. While individuals aged 25-44 years had the highest incidence of HIV, youth aged 13-24 years comprised more than half of all new diagnoses [1].

    Pre-exposure prophylaxis (PrEP) is an effective daily prevention method and should be considered for use by HIV-negative populations at higher risk of HIV [3]. When taken as prescribed, PrEP can offer almost complete protection against HIV; however, previous studies have shown that PrEP uptake is lower in higher risk populations [4,5]. Barriers to PrEP uptake include limited awareness and limited access as a result of PrEP marketing and additional socioeconomic challenges, respectively [5].

    Moreover, adolescents and young adults are major consumers of technology and often rely on the internet for health information [1,6-8]. Since adolescents and young adults may be consulting the internet instead of health care providers for HIV prevention information, it is critical that health information on the web is accurate and readable. To the authors’ knowledge, there is limited research assessing the quality of PrEP information available on the web. Therefore, the objectives of this study are to determine if accurate and readable PrEP information can be found using web search engines and virtual assistants. Additionally, this study aimed to assess how PrEP information found via both methods compared among and between search engines and virtual assistants.


    Ethical Considerations

    The institutional review board at Children’s Hospital of Philadelphia determined that approval was not necessary for this study as it did not contain human subjects.

    Developing a Framework for Evaluating PrEP Internet Content

    Common PrEP information themes were determined for future searches within the United States using English only: PrEP basics, PrEP access, and PrEP use (Textbox 1). A total of 50 questions were compiled through an iterative process with an expert panel of adolescent medicine researchers and physicians who work with adolescents and young adults at high risk of contracting HIV as well as youth living with HIV. Items were narrowed down to 23 questions designed to address concerns adolescents and young adults may have about general use, privacy, and confidentiality (Table 1).

    Textbox 1. PrEP categories.

    Pre-exposure prophylaxis (PrEP) basics

    • The questions in this category aim to find general information about PrEP for HIV prevention to introduce and educate potential users about this intervention method.

    PrEP access

    • The questions in this category aim to find information about ways users can access PrEP in addition to information about privacy concerns.

    PrEP use

    • The questions in this category aim to find information regarding the proper use of PrEP so that potential users can receive the maximum benefits.
    Table 1. PrEP internet search questions.
    QuestionCategory
    What is PrEPa?Basics
    What kinds of PrEP are available?Basics
    What is Truvada?Basics
    What is Descovy?Basics
    What is the difference between Truvada and Descovy?Basics
    Does PrEP protect against other sexually transmitted infections/sexually transmitted diseases?bBasics
    Should I take PrEP?Basics
    What are the side effects of taking PrEP?Basics, use
    Will PrEP make me feel sick?Basics
    How does PrEP work?Basics, use
    What is the difference between PEPc and PrEP?Basics
    How effective is PrEP?Basics, use
    Where can I find PrEP?Access
    How can I get PrEP?Access
    Do I need insurance to get PrEP?Access
    Do I need my parents’ permission to get PrEP?Access
    Will my parents know that I’m using PrEP if I’m on their insurance?Access
    How much does PrEP cost?Access
    How do I take PrEP?Use
    When can I stop taking PrEP?Use
    How long does it take for PrEP to start working?Use
    What happens if I miss a dose of PrEP?Use, basics
    How do I refill my PrEP prescription?Use, access

    aPrEP: pre-exposure prophylaxis.

    bThis question was asked using both “sexually transmitted infections” and “sexually transmitted diseases” due to the recent change in terminology.

    cPEP: postexposure prophylaxis.

    Data Collection: Web Search Engines and Virtual Assistants

    Questions were entered into search engines—Ask.com, Bing, Google, and Yahoo—using a private Mozilla Firefox browser and verbally posed to virtual assistants—Amazon Alexa, Microsoft Cortana, Google Assistant, and Apple Siri—using new user accounts apart from Apple Siri where a pre-existing account with cleared browsing history was used. Questions were asked to search engines and virtual assistants on separate days in March 2021 with a maximum of 10 times for virtual assistants to generate a response. Excluding advertised content, information from the first three web search engine results and virtual assistant nonverbal results were transcribed as listed on the website and recorded in corresponding Excel (Microsoft Corporation) sheets for future analysis. Verbal responses from each virtual assistant were recorded, transcribed, and saved in Excel.

    Data Analyses: Scale Development, Rating System for Information Accuracy, and Readability Assessment

    A six-tier scale was created and reviewed by an adolescent medicine physician and a researcher to code example content (Table 2). The examples were used as content validation to ensure that results were coded using the same criteria. Results received two scores from two different coders who assigned final scores after reaching a consensus. Final scores were then used to calculate an average quality score for each web-based resource in each category. The data collected were also entered into a web-based literacy measurement tool [9] to assess readability using the Flesch-Kincaid Grade Level Scale [10].

    Table 2. PrEP Information Quality Rating Scale.
    Rating numberMeaningExample content (question: what is PrEPa?)
    0The resulting answer did not include any information about PrEP.“HIV (human immunodeficiency virus) is a virus that attacks the body’s immune system. There is currently no effective cure. Once people get HIV, they have it for life” [11]
    1The resulting answer included information about PrEP that was completely inaccurate.“PrEP is a medication taken only right before sex in order to prevent HIV. While taking PrEP, individuals do not need to use condoms” [12]
    2The resulting information about PrEP was partially correct but contained some errors/misleading information. The source providing information was unreliable.“PrEP is a pill taken before sex, so it is pre-exposure. Prophylaxis means to prevent infection. So you can use PrEP to greatly reduce the risk of becoming HIV positive” [13]
    3The resulting information was accurate but lacked many important details.“Pre-exposure Prophylaxis, also known as PrEP, is when people take medicine to lower their risk of getting HIV” [14]
    4The resulting information was detailed, accurate, and from a reliable source. However, the information included was outdated.“PrEP is an experimental approach to HIV prevention and consists of antiretroviral drugs to be taken before potential HIV exposure in order to reduce the risk of HIV infection and continued during periods of risk” [15]
    5The resulting information was detailed, accurate, and from a reliable source. Additionally, the result included the latest information available.“Pre-exposure prophylaxis (or PrEP) is a way for people who do not have HIV but who are at very high risk of getting HIV to prevent HIV infection by taking a pill every day. The pill (brand name Truvada) contains two medicines (tenofovir and emtricitabine) that are used in combination with other medicines to treat HIV. When someone is exposed to HIV through sex or injection drug use, these medicines can work to keep the virus from establishing a permanent infection. When taken daily, PrEP is highly effective for preventing HIV. Studies have shown that PrEP reduces the risk of getting HIV from sex by about 99% when taken daily. Among people who inject drugs, PrEP reduces the risk of getting HIV by at least 74% when taken daily. PrEP is much less effective if it is not taken consistently. As PrEP only protects against HIV, condoms are important for the protection against other STDs. Condoms are also an important prevention strategy if PrEP is not taken consistently.” [16]

    aPrEP: pre-exposure prophylaxis.


    Web-Based PrEP Information Quality

    From all search engines and virtual assistants, 422 results were compiled and assigned a score from the PrEP Information Quality Scale (Table 2). The results show that Google search engine and Google Assistant more frequently provided higher quality information than resources from other companies, and that search engines provided higher quality information than virtual assistants (Table 3). Google search engine provided more comprehensive results directing potential users to websites with information on the ideal candidates, proper use, and access. However, Ask.com often provided the lowest quality information directing users to websites containing no information about PrEP. Bing and Yahoo provided information quality between Google and Ask.com.

    Table 3. Search engines’ and virtual assistants’ average PrEP information quality scoresa.
    ResourcePrEPb basicsPrEP accessPrEP use
    Search engines

    		Ask.com1.80.80.5

    		Bing3.72.73.1

    		Google3.43.73.5

    		Yahoo3.42.82.9
    Virtual assistants

    		Amazon Alexa1.10.91.4

    		Apple Siri2.41.83.3

    		Google Assistant3.23.32.8

    		Microsoft Cortana2.32.52.0

    aPrEP information score meanings: 0=irrelevant, 1=inaccurate, 2=partially correct/misleading, 3=accurate but missing important details, 4=accurate but outdated, 5=accurate and current.

    bPrEP: pre-exposure prophylaxis.

    The following responses were given to the question “How much does PrEP cost?”

    ...A month’s supply of Truvada is nearly $2,000 without insurance. Most private health insurance companies, Medicare, and Medicaid will cover the cost...
    [WebMD via Google]
    Launching a startup...
    [Ask.com]

    Within virtual assistants, Google Assistant typically provided more informative results when compared to the other virtual assistants selecting relevant text from websites to respond to posed questions. Amazon Alexa generally provided the lowest quality information with irrelevant responses including “Sorry, I don’t know that one.” Microsoft Cortana and Apple Siri provided information quality between Google Assistant and Amazon Alexa.

    The following responses were given to the question “What is the difference between Truvada and Descovy?”

    There are now two medications approved by the U.S Food and Drug Administration (FDA): Truvada and Descovy...which medication might be right for you, take a look at the similarities and differences between the two medications below...
    [SFAF.org via Google Assistant]
    Pharmaceutical product. Star Trek Discovery...
    [Amazon Alexa]

    Web-Based PrEP Information Readability

    The readability assessment using the Flesch-Kincaid Grade Level scale revealed that information from web search engines produced by questions in all three categories requires a literacy level that aligns with that of an eighth grade, ninth grade, or 10th grade student, while virtual assistant results require literacy levels ranging from a second grade to eighth grade student in the United States (Table 4). The required literacy level of information produced by search engines and virtual assistants appears to correlate with the quality of information produced by each, as the search engines more frequently provided information of higher quality than the virtual assistants.

    Table 4. Search engines’ and virtual assistants’ average PrEP information readability scoresa.
    ResourcePrEPb basicsPrEP accessPrEP use
    Search engines

    		Ask.com8.39.28.3

    		Bing10.28.19.1

    		Google9.39.68.6

    		Yahoo9.78.19.2
    Virtual assistants

    		Amazon Alexa7.02.75.4

    		Apple Siri8.44.87.1

    		Google Assistant7.48.36.9

    		Microsoft Cortana7.06.47.7

    aPrEP information readability score formula: grade level = 0.37 (words/sentence) + 5.84 (syllables/word) – 15.59.

    bPrEP: pre-exposure prophylaxis.


    Principal Results

    Google search engine and Google Assistant produced higher quality PrEP information more frequently than the other web-based resources used. Authors found resulting answers to PrEP questions that conflicted with each other within a single search engine/virtual assistant, which may lead to incorrect use of PrEP and ultimately reduced effectiveness. Additionally, the resulting information generally was presented in language between a seventh and 10th grade reading level, therefore, often exceeding the average reading level of adolescents and young adults in the United States [17].

    Comparison With Prior Work

    To the authors’ knowledge, there are no other studies that have specifically evaluated the quality of PrEP information for HIV prevention produced by virtual assistants or the readability of health information found on the web. However, a New Zealand study titled “In Bed with Siri and Google Assistant: A Comparison of Sexual Health Advice” used Google search engine, Google Assistant, and Apple Siri to assess the quality of sexual health advice received from each source. The results of this study are consistent with researchers’ findings that Google search engine had the best responses, followed by Google Assistant then Apple Siri, when asked questions about sex and sexual health [18].

    Limitations

    The study limitations include the variability of results produced by search engine and virtual assistant algorithms. To account for the possibility of different top results for the same questions, searches were performed on a single day for all search engines and on a separate day within the same week for all virtual assistants. Future studies may consider conducting a time-based analysis where searches are completed at set time intervals over 1 year or more to compare top results for the same questions over time. Another limitation is the condensed list of questions. Future studies should include youth perspectives on what questions adolescents and young adults would ask in addition to provider opinions. Additionally, the study was conducted in English, which may exclude individuals whose primary language is not English. Furthermore, at the time of this study, injectable PrEP had not yet been approved and are not included in the search results.

    Conclusions

    Adolescents and young adults may turn to technology due to discomfort discussing their sexual health with providers, yet we found that there is much room for improvement in the quality and readability of educational information on PrEP through web-based resources.

    Acknowledgments

    This research was directly supported by grants from the National Institute of Mental Health P30MH097488 (principal investigator: Dwight Evans) and K23MH119976-01A1 (principal investigator: SW) and the Leadership Education in Adolescent and Young Adult Health program.

    Conflicts of Interest

    None declared.

    1. HIV surveillance report, 2018 (updated). Centers for Disease Control and Prevention. 2020. URL: https:/​/www.​cdc.gov/​hiv/​pdf/​library/​reports/​surveillance/​cdc-hiv-surveillance-report-2018-updated-vol-31.​pdf [accessed 2023-06-27]
    2. HIV surveillance report. Centers for Disease Control and Prevention. 2020. URL: https:/​/www.​cdc.gov/​hiv/​pdf/​library/​reports/​surveillance/​cdc-hiv-surveillance-report-2018-preliminary-vol-30-infographic.​pdf [accessed 2023-06-27]
    3. Pre-exposure prophylaxis (PrEP). HIVinfo. 2020. URL: https://hivinfo.nih.gov/understanding-hiv/fact-sheets/pre-exposure-prophylaxis-prep [accessed 2021-01-14]
    4. PrEP effectiveness. Centers for Disease Control and Prevention. 2020. URL: https://www.cdc.gov/hiv/basics/prep/prep-effectiveness.html [accessed 2021-01-19]
    5. Wood SM, Lee S, Barg FK, Castillo M, Dowshen N. Young transgender women's attitudes toward HIV pre-exposure prophylaxis. J Adolesc Health. May 2017;60(5):549-555. [FREE Full text] [CrossRef] [Medline]
    6. Hargittai E, Hinnant A. Digital inequality: differences in young adults' use of the internet. Commun Res. Aug 04, 2008;35(5):602-621. [CrossRef]
    7. Escoffery C, Miner KR, Adame DD, Butler S, McCormick L, Mendell E. Internet use for health information among college students. J Am Coll Health. Aug 06, 2005;53(4):183-188. [CrossRef] [Medline]
    8. Anderson M, Jiang J. Teens, social media and technology 2018. Pew Research Center. 2018. URL: https:/​/www.​pewresearch.org/​internet/​wp-content/​uploads/​sites/​9/​2018/​05/​PI_2018.​05.​31_TeensTech_FINAL.​pdf [accessed 2021-02-11]
    9. Kydala Publishing Inc. StoryToolz. URL: https://storytoolz.com [accessed 2021-03-17]
    10. Kincaid JP, Fishbourne Jr. RP, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) for Navy enlisted personnel. Institute for Simulation and Training. Jan 01, 1975;56:i-39. [CrossRef]
    11. About HIV. Centers for Disease Control and Prevention. 2020. URL: https://tinyurl.com/3c7nmxsr [accessed 2021-02-01]
    12. PrEP myths. Houston Health Department. 2019. URL: https://houstoniamlife.com/prep/prep-myths/ [accessed 2021-02-01]
    13. What is PrEP? HIV i-Base. 2019. URL: https://i-base.info/guides/prep/what-is-prep [accessed 2021-02-01]
    14. What is PrEP? National Women’s Health Network. 2018. URL: https://nwhn.org/what-is-prep/ [accessed 2021-02-01]
    15. Naswa S, Marfatia YS. Pre-exposure prophylaxis of HIV. Indian J Sex Transm Dis AIDS. Jan 2011;32(1):1-8. [CrossRef] [Medline]
    16. The basics - PrEP. Prepfacts. 2020. URL: https://www.cdc.gov/hiv/risk/prep/index.html#print [accessed 2021-04-06]
    17. Marchand L. What is readability and why should content editors care about it? Center for Plain Language. 2017. URL: https://tinyurl.com/3vykwjze [accessed 2021-04-02]
    18. Wilson N, MacDonald EJ, Mansoor OD, Morgan J. In bed with Siri and Google Assistant: a comparison of sexual health advice. BMJ. Dec 13, 2017;359:j5635-j5642. [CrossRef] [Medline]

    ‎
    CDC: Centers for Disease Control and Prevention
    PrEP: pre-exposure prophylaxis

    Edited by S Badawy; submitted 09.09.22; peer-reviewed by U Belkind; comments to author 04.03.23; revised version received 31.03.23; accepted 01.04.23; published 18.07.23.

    Copyright

    ©Kaja Darien, Susan Lee, Kayla Knowles, Sarah Wood, Miriam D Langer, Nellie Lazar, Nadia Dowshen. Originally published in JMIR Pediatrics and Parenting (https://pediatrics.jmir.org), 18.07.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Pediatrics and Parenting, is properly cited. The complete bibliographic information, a link to the original publication on https://pediatrics.jmir.org, as well as this copyright and license information must be included.