Evaluation of mobile health applications for cervical cancer in the digital marketplace

Jakkapop Kanjak; Naratassapol Likitdee; Chumnan Kietpeerakool; Amornrat Temtanakitpaisan

doi:10.5468/ogs.22037

Obstet Gynecol Sci > Volume 65(3); 2022 > Article

Kanjak, Likitdee, Kietpeerakool, and Temtanakitpaisan: Evaluation of mobile health applications for cervical cancer in the digital marketplace

Original Article

Gynecologic Oncology

Obstetrics & Gynecology Science 2022;65(3):244-255.

Published online: April 5, 2022

DOI: https://doi.org/10.5468/ogs.22037

Evaluation of mobile health applications for cervical cancer in the digital marketplace

Jakkapop Kanjak, MD, Naratassapol Likitdee, MD

, Chumnan Kietpeerakool, MD, Amornrat Temtanakitpaisan, MD

Department of Obstetrics and Gynecology, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand

Corresponding author: Naratassapol Likitdee, MD, Department of Obstetrics and Gynecology, Faculty of Medicine, Khon Kaen University, 123 Mittraphap Road, Muang District, Khon Kaen 40002, Thailand, E-mail: polli@kku.ac.th

Received January 19, 2022 Revised February 17, 2022 Accepted February 28, 2022

Articles published in Obstet Gynecol Sci are open-access, distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Objective

To assess the quality of mobile health (mHealth) applications (apps) for cervical cancer using the mobile app rating scale (MARS), APPLICATIONS scoring system, and app rating using specific statements.

Methods

We searched for cervical cancer apps on two major mobile operating systems (Google Play Store and Apple iTunes Store) in March 2021. Eligible apps were downloaded and assessed for quality by two independent reviewers using multimodal assessment tools.

Results

The overall quality of the MARS score was 2.61±0.795. The highest scoring app was “The American Society for Colposcopy and Cervical Pathology (ASCCP) Management Guidelines” (3.98). Overall, apps scored highest in the functionality domain, followed by information, engagement, and aesthetics domains. The mean±standard deviation of the APPLICATIONS scoring system was 8.50±1.712. The highest-rated apps were “ASCCP Management Guidelines,” “The British Society for Colposcopy and Cervical Pathology (BSCCP),” and “Cervical Cancer Guide.” Apps scored the highest in the paid subscription and price domains. By contrast, apps scored poorly in the text search, literature, and subjective presentation domains. Concerning app content, many apps infrequently provided misconceptions regarding cervical cancer. The apps’ rating using specific statements was 7.81±4.562.

Conclusion

Overall, the apps analyzed using the MARS and APPLICATIONS scoring systems demonstrated above-average quality. However, there is a need to improve the essential information conveyed by these applications. Moreover, the assessment tools have influenced different app quality rating results, confirming the lack of standardized quality assessment tools for mHealth apps.

Keywords: Cervix uteri; Neoplasms; Health communication; Health promotion; Mobile applications

Introduction

Cervical cancer remains a significant public health problem in low-resource settings [1]. It is the 2nd leading cause of cancer in women in Thailand, ranking after breast cancer [2]. The level of knowledge of cervical cancer is associated with the success of cervical cancer programs [3-5]. A recent nationwide social media survey in Thailand found that nearly half of Thai women have insufficient knowledge regarding cervical cancer, indicating the need for further health education intervention [6].

Mobile health (mHealth) applications (apps) have been acknowledged as effective tools for improving population-level health outcomes. In a recent systematic review undertaken to assess the use of mHealth apps in low- and middle-income countries, this health technology intervention improved various health outcomes. The most prominent health outcomes that improved with mHealth apps were communicable diseases and maternal health. These findings thus support the use of mHealth apps for health promotion in developing countries [7].

mHealth apps for cancer information are increasingly used as health education interventions for individuals. Many apps for cancer information provide general knowledge about the incidence, risk factors, signs and symptoms, diagnosis, treatment, and screening methods. Thus, they could facilitate cancer awareness in women and encourage participation in prevention and treatment programs [8-11]. Most healthcare providers have a positive attitude toward using oncological apps [11].

Significant numbers of gynecologic cancer apps, however, are not up-to-date and some of the contents are unreliable. In addition, almost all apps have no stakeholders involved in their development [12-15]. A previous study noted that only 1.5% (11 of 748) of gynecologic cancer apps were found to be both potentially helpful and accurate [16]. Owing to concerns about the quality of mHealth apps, the US Food and Drug Administration (FDA) has launched policies regarding the development of mHealth apps [17]. However, many apps are neither reviewed nor certified before being released to the public.

High-quality mHealth apps can serve as novel interventions to combat barriers that limit effective cervical cancer prevention and treatment by improving cancer awareness and changing behavior in healthy women and cancer survivors, such as enhancing the uptake of cervical cancer screening and increasing adherence to follow-up [18-21]. Several studies have described tools for assessing the quality of mHealth apps in cancer, such as the mobile app rating scale (MARS) [22-24], APPLICATIONS scoring system [16,25], and digital health scorecard approach [26]. A recent systematic review identified the standard domains of quality assessment criteria for mHealth apps and summarized 15 domains for researchers, developers, and users to assess the quality of the apps themselves [27]. Our study was conducted to assess the quality of cervical cancer apps using multimodal assessment tools (MARS, APPLICATIONS scoring system, and app rating by specific statements).

Materials and methods

1. Study design and setting

We performed a cross-sectional study to assess the quality of mHealth apps providing cervical cancer information available on two major mobile operating systems: Google Play Store (Android) and iTunes Store (iOS) [28]. Our study was conducted based on the five recommended steps for the quality assessment of mHealth apps [27].

2. Search strategy

Two authors searched mobile applications using the search bar on Google Play Store and iOS to identify cervical cancer-related apps. One author searched the Google Play Store, and the other searched iOS between March 1 and March 22, 2021. The following Medical Subject Heading (MeSH) terms appearing in MEDLINE/PubMed databases were used for the search: “cervical cancer” (mareng pak mdluk; cervical cancer), “cervix cancer,” “cervical tumor,” “cervix tumor,” “cervical mass,” “cervix mass,” “cervical malignancy,” “cervix malignancy,” “cervical neoplasia,” and “cervix neoplasia.” Any new apps identified by these MeSH term searches were added to the list of apps for each mobile operating system.

3. Selection criteria

After duplicate apps were manually removed, a complete list of apps from each mobile operating system was initially screened based on their title and description. Only apps related to cervical cancer available in English or Thai, free apps (no cost), and apps for a fee were included in the study. Apps were excluded if 1) they had no scientific content, 2) they were not specific to cervical cancer, or 3) they were non-oncologic apps, for example, horoscope apps. Apps that met the selection criteria were downloaded using either an ASUS ZenPhone Max Pro M1 mobile phone for the Google Play Store or an iPhone 6S mobile phone for the iTunes store for further assessment. Authors JK and NL assessed the accuracy of eligible apps to ensure that they were appropriate. Disagreements between the two reviewers were resolved through consensus-based discussion. The selection process is illustrated in Fig. 1.

4. Data collection and app characteristics

Basic information on the apps, including the name of the app, name of the developer or seller, platform (Android or iOS), price, number of downloads (Android only), date of the last update, user rating, app store category, and language, was recorded using a Microsoft Excel spreadsheet (Office 365) for further analysis.

5. Apps evaluation

The included apps were independently reviewed by two authors (JK and NL) between April 5, 2021 and May 24, 2021. Each app was initially analyzed offline to determine function without internet connectivity. Apps that required payment for premium services were paid for to evaluate the full functionality of the app. Both reviewers used the apps for 7 days before performing quality assessment according to the recommended steps for research on the quality assessment of mHealth apps [27]. Each app was assessed using MARS [24], followed by the APPLICATIONS scoring system [16] and app ratings by specific questions. Before conducting the evaluation, the reviewers underwent MARS training using a MARS training video available on YouTube [29]. Shared understanding of MARS, APPLICATIONS scoring system items, and app ratings by specific statements was followed by an independent review of a few apps and discussion of scores to assess consensus. Any disagreements were discussed with a third reviewer (CK or AT).

6. MARS evaluation

The MARS evaluation tool consists of 19 items and four questions, as shown in Table 1 [24,29]. The first 19 items were designed to analyze the four domains of the objective quality section: 1) engagement, 2) functionality, 3) esthetics, and 4) information. The score for each item ranges from 1 to 5 (1=inadequate; 2=poor; 3=acceptable; 4=good; 5=excellent). The mean scores for each domain were calculated, and the total sum of the objective domains was divided by 4 to develop the average objective quality score of the apps. The fifth domain of the MARS consists of four questions and is designed to analyze the subjective quality section. The total MARS score was calculated using the objective and subjective quality scores to describe the overall quality of the app. The final score was calculated by two authors, as shown in Table 1.

7. APPLICATIONS scoring system evaluation

The APPLICATIONS scoring system was developed based on existing literature [16,25]. This assessment tool includes 10 objective components and two subjective components, as shown in Table 2. The comprehensiveness score was dependent on the completeness of the topic in the app content. The app received 1 point for each criterion, contributing to a total of 3 points. Subjective components, including ease of navigation and presentation, were evaluated. The score for each item ranges from 1 to 5 (1=inadequate; 2=poor; 3=acceptable; 4=good; 5=excellent). An average score was calculated by two authors; less than three received no points, and three or higher received 1 point with an average score. The total score is calculated as the sum of the objective and subjective component scores.

8. App rating using specific statements

To provide a more specific assessment, we developed 16 statements regarding misconceptions about cervical cancer based on a previous study, as shown in Table 3 [6,30]. The app with content that matches a specific statement receives 1 point for each content that matches the specific statement. The maximum score for the app rating was 16.

9. Statistical methods

Descriptive statistics, including numbers and percentages, were used to categorize the characteristics of apps, such as platforms (Android or iOS), free or paid, user rating, app store category, and language. App evaluations were calculated and presented as raw and mean scores, with a standard deviation (SD) for each assessment tool. All statistical analyses were performed using Microsoft Excel (Office 365).

Results

1. App identification

Seven hundred sixty-three apps were identified (727 from Google Play Store and 36 from the iTunes Store). Four hundred and fifty-nine applications were excluded owing to duplication. After screening the app names and descriptions, 265 were excluded. The remaining 39 apps were downloaded and assessed for their eligibility. After a detailed evaluation, 23 apps were excluded, leaving 16 included in our study (Fig. 1).

2. App characteristics

Table 4 summarizes the characteristics of the apps included. Of all the apps, 16 (100%) were available on the Android platform, and seven (43.8%) were available on both the Android and iOS platforms. The most significant number of apps were classified as medicine, followed by health, fitness, and entertainment. All apps were in English. Only one app was in both English and Thai. Most apps (87.5%) had been updated over a 3-year period (2019-2021), and 14 (87.5%) were free to download. Only two apps required payment, with costs ranging from $3.49 to $9.99. The number of app downloads ranged from 1 to 100,000. The National Comprehensive Cancer Network (NCCN) Guidelines^®, produced by this scientific organization, was the only high-use app, with >100,000 downloads. The average star ratings on the Android and iOS platforms range from 0 to 4.8.

3. App evaluation using MARS

Table 1 shows the MARS scores of the 16 apps stratified by quality. In terms of objective quality, the mean objective quality score was 2.92±0.57 out of 5, with the highest scoring app being “The American Society for Colposcopy and Cervical Pathology (ASCCP) Management Guidelines” (3.96). Overall, apps scored highest in the functionality domain (3.45±0.76), followed by information (3.40±0.76), engagement (2.45±0.71), and aesthetic (2.42±0.84) domains. In terms of the subjective quality section, the mean subjective quality score was 2.28±1.10 out of 5, with the highest scoring apps being “ASCCP Management Guidelines” (4.00) and “NCCN Guidelines^®” (4.00). The overall quality MARS score was 2.61±0.79 out of 5. The highest scoring app was “ASCCP Management Guidelines” (3.98), followed by “MyPap” (3.58), “FIGO Gyn Cancer Management” (3.54), and “NCCN Guidelines^®” (3.54).

4. App evaluation using the APPLICATIONS scoring system

The APPLICATION scores of the 16 apps are listed in Table 2. The mean±SD was 8.50±1.71 of 16. The highest scoring apps were “ASCCP Management Guidelines,” “The British Society for Colposcopy and Cervical Pathology (BSCCP),” and “Cervical Cancer Guide,” each receiving 11 points, closely followed by “NCCN Guidelines^®” and “Cervical Cancer Symptoms,” each receiving 10 points. The lowest app received 6 points from a total of 16 points. Overall, apps scored highest in the paid subscription and price domains, because almost all apps were free to download. By contrast, apps scored poorly in the text search, literature, and subjective presentation domains. The app comprehensiveness mean score±SD was 1.94±1.00 out of 3.

5. App rating using specific statements

The app “Cervical Cancer,” developed by Anastore, has content that most highly matches misconceptions about cervical cancer (14 of 16 statements), followed by “NCCN Guidelines^®” (13 of 16 statements). The mean score of all apps was 7.81±4.56 out of 16. Most apps (12 of 16) provide information that human papillomavirus (HPV) causes cervical cancer, and that smoking increases the risk of cervical cancer. However, none of the apps mentioned that HPV infection could not be prevented by vaginal douching after intercourse. Only one app provided information indicating that HPV smear results cannot be obtained immediately after vaginal examination (Table 3).

6. Comparison of app scores using different assessment tools

The top three apps identified by each assessment tool differed, as shown in Table 5. However, some apps, such as “NCCN Guidelines^®,” were rated as high-quality by all three assessment tools, followed by “ASCCP Management Guidelines,” which were rated as high-quality apps by two assessment tools (MARS and APPLICATIONS scoring system). Some apps were rated as high-quality apps by one assessment tool but were not ranked as high-quality apps by other assessment tools, such as “MyPap,” “FIGO Gyn Cancer Management,” “Cervical Cancer Symptoms,” “BSCCP,” “Cervical Cancer” developed by Fumo, and “Cervical Cancer” developed by Anastore. An app’s average star rating did not seem to correlate with the ratings of other assessment tools.

Discussion

Our study noted that cervical cancer apps available in the digital marketplace demonstrate above-average quality. However, the apps scored poorly in app-specific ratings using specific statements, indicating a need for improvement in terms of the essential information conveyed by these apps. Moreover, the type of assessment tool used influenced the different app quality rating results. Many assessment tools can be used to determine the quality of mHealth apps for cancer. Therefore, our study applied the MARS and APPLICATIONS scoring systems to evaluate the quality of the apps. The MARS and APPLICATIONS scoring systems have been frequently used to assess mHealth app quality over the past 6 years [16,22-25]. However, no studies have used both scoring systems and compared their results. The tools we used had domain criteria for evaluation that matched a recent systematic review of the quality assessment criteria for mHealth apps, which suggested criteria for future research on the quality of mHealth apps [27].

All apps in our study were available on the Google Play Store, with a more significant percentage available than the iTunes Store. Our findings were consistent with the results of previous studies, which found that more apps were available on the Google Play Store than on the iTunes Store [9,10,13]. A recent study found that up-to-date apps are uncommon [13,22]. Contrary to our findings, most cervical cancer apps (89.5%) were up-to-date over the last 3 years.

Our study showed a mean score of 2.61±0.79 out of 5 for the MARS overall quality score of the apps, which is similar to a previous study that showed a mean score of 2.98 out of 5 for the mHealth app for genitourinary tumors [22]. However, these scores were lower than for the evaluation of mobile apps to track patient-reported outcomes for cancer patients (3.50 of 5) [31], and the evaluation of apps to support medication adherence and symptom management in cancer patients showed a mean score ranging from 2.8 to 4.3 out of 5 [23]. One study reported a significantly lower mean quality score (1.96/5) for prostate cancer, breast cancer, and colorectal cancer apps [12].

We found that apps scored poorly in the MARS engagement domain, in that most apps are unable to entertain and have no interactive function with the app’s user. In contrast, apps scored highest in the functionality domain compared to other domains. These scores are consistent with the results of 2 previous studies [22,23].

One study omitted the subjective MARS quality score from evaluation because this domain criterion might impose a risk for bias [23]. However, our study included all MARS domains for evaluation. The subjective quality domain is displayed separately in Table 1, before the averaging of the overall MARS quality score in the final column.

Our study showed a significantly lower mean APPLICATIONS score (8.50±1.71 of 16) compared with a study of gynecologic cancer apps [16], which reported a mean score of 10±2.30 out of 16. However, the previous study was not specific to evaluating cervical cancer apps, thus limiting direct comparison of the results.

Cervical cancer is a significant health problem in developing countries [32]. However, some app content may not contain essential information that focuses on users in developing countries. Therefore, we developed a specific tool using misconceptions about cervical cancer from previous studies as a tool for assessment to ensure that apps provide this information [6,30].

The number of app downloads and average star ratings in digital marketplaces may not guarantee the quality of the apps. Only one app in our study (NCCN Guidelines^®) was rated as high quality by all three assessment tools. However, the top three apps as determined by each assessment tool were different, which confirms the lack of standardized quality assessment tools for mHealth apps.

Most developers are commercial companies, and scientific organizations or healthcare providers have developed only a few apps (NCCN Guidelines and ASCCP Management Guidelines). According to our results, apps developed by scientific organizations are associated with high-quality scores. Incorporating healthcare professionals as partners or stakeholders during the development process may strengthen the quality of the apps.

Our study is the first to perform a comprehensive assessment of cervical cancer apps using multimodality assessment tools and compare the results from each assessment tool. However, this study has some limitations. First, this study did not include apps in languages other than English and Thai, which limits the generalizability of our findings. Second, new apps are being developed and updated all the time in the digital marketplace. There is a possibility that some apps may have been launched after our research was completed and thus were not included in this study. Finally, this study focused on mHealth apps and we did not therefore include websites or social media content, which have become common sources of consumer information.

Cervical cancer apps that are now available in the digital marketplace are of above-average quality. However, there is a need for improvement in terms of the information conveyed by these applications. Moreover, the assessment tools influenced different app quality rating results, confirming the lack of standardized quality assessment tools for mHealth apps.

Acknowledgments

The abstract of this paper was presented at the 7th Biennial Meeting of the Asian Society of Gynecologic Oncology 2021 Conference, titled “Evaluation of Mobile Health Applications for Cervical Cancer in the Digital Marketplace,” as an e-poster presentation. The authors declare no conflicts of interest.

Notes

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Ethical approval

Our study was exempt from ethical approval by the Ethics Committee of Human Research, Khon Kaen University, as it did not involve human subjects.

Patient consent

Written informed consent and the use of images from patients are not required for the publication.

Funding information

This study was supported by a grant from the Faculty of Medicine of Khon Kaen University in Thailand (Grant Number IN64232).

Fig. 1

Flow diagram of the process utilized to identify cervical cancer apps for evaluation. Apps, applications; iOS, an operation system used for mobile devices manufactured by Apple Inc.

Table 1

Mobile app rating scale of cervical cancer applications

App name	Objective quality section				Objective quality score (5 points)	Subjective quality score (5 points)	Overall quality MARS score (5 points)
App name	Engagement (5 points)	Functionality (5 points)	Aesthetics (5 points)	Information (5 points)	Objective quality score (5 points)	Subjective quality score (5 points)	Overall quality MARS score (5 points)
1. Mareng Pak Mdluk^a)	2.20	4.00	2.33	3.29	2.95	2.75	2.85
2. Cervical cancer guide	2.20	4.25	2.00	3.14	2.90	2.25	2.58
3. Cervical Cancer (Bedieman)	1.60	3.75	1.00	3.29	2.41	2.25	2.33
4. Cervical Cancer (Nature Healthy Care)	1.80	3.75	2.67	3.14	2.84	1.75	2.30
5. Cervical Cancer (Fumo)	1.80	4.25	1.67	3.29	2.75	2.25	2.50
6. MyPap	3.40	4.25	2.67	3.29	3.40	3.75	3.58
7. Scanvix	2.40	3.00	2.33	2.85	2.65	1.25	1.95
8. Cervical Cancer Symptoms	2.20	2.75	2.00	2.71	2.42	1.50	1.96
9. Cervical cancer (Anastore)	2.40	2.00	1.00	2.71	2.03	1.00	1.52
10. TNM Cancer Staging Calculator	3.00	4.25	3.67	3.43	3.59	3.25	3.42
11. BSCCP	3.00	3.00	3.33	3.33	3.08	1.25	2.17
12. Cancer Genetics	2.40	3.50	2.33	3.43	2.92	1.00	1.96
13. FIGO Gyn Cancer Management	4.20	3.50	3.00	4.57	3.82	3.25	3.54
14. NCCN Guidelines^®	1.80	3.00	2.67	4.86	3.08	4.00	3.54
15. Cervical Cancer (Personal Remedies LLC)	1.80	2.00	2.00	2.14	1.99	1.00	1.50
16. ASCCP Management Guidelines	3.00	4.00	4.00	4.86	3.96	4.00	3.98
Mean±standard deviation	2.45±0.71	3.45±0.76	2.42±0.84	3.40±0.76	2.92±0.57	2.28±1.10	2.61±0.79

App, application; MARS, mobile app rating scale; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

^a) Mareng Pak Mdluk” means cervical cancer in Thai language.

Table 2

APPLICATIONS score of cervical cancer apps

App name	APPLICATION scoring system												Total score (16 points)
App name	A	P	P	L	I	C	A	T	I	O	N	S	Total score (16 points)
1. Mareng Pak Mdluk^a)	3	1	1	0	1	0	0	0	0	1	1	1	9
2. Cervical cancer guide	3	1	1	0	1	1	1	0	0	1	1	1	11
3. Cervical Cancer (Bedieman)	3	1	1	0	1	1	1	0	0	0	1	0	9
4. Cervical Cancer (Nature Healthy Care)	2	1	1	0	1	0	0	0	0	1	1	0	7
5. Cervical Cancer (Fumo)	3	1	1	0	1	1	0	0	0	0	1	0	8
6. MyPap	1	1	1	0	0	1	1	0	0	1	1	0	7
7. Scanvix	2	1	1	0	1	1	1	0	0	0	0	0	7
8. Cervical Cancer Symptoms	3	1	1	0	1	1	0	0	0	1	1	1	10
9. Cervical cancer (Anastore)	1	1	1	1	0	0	0	0	0	1	1	0	6
10. TNM Cancer Staging Calculator	1	1	1	0	1	1	0	0	1	0	1	1	8
11. BSCCP	2	1	0	0	1	1	1	1	1	1	1	1	11
12. Cancer Genetics	0	1	1	1	1	0	1	0	1	0	0	0	6
13. FIGO Gyn Cancer Management	1	1	1	1	0	0	1	0	1	1	1	0	8
14. NCCN Guidelines^®	3	1	1	1	1	0	1	1	1	0	0	0	10
15. Cervical Cancer (Personal Remedies LLC)	1	0	1	1	1	0	1	0	1	2	0	0	8
16. ASCCP Management Guidelines	2	0	1	1	1	0	1	0	1	2	1	1	11
Mean±standard deviation	1.94±1.00	0.88±0.34	0.94±0.25	0.38±0.50	0.81±0.40	0.50±0.52	0.63±0.50	0.13±0.34	0.44±0.51	0.75±0.68	0.75±0.45	0.38±0.50	8.50±1.71

App, application; A, app comprehensiveness (3 points); P, price (1 point); P, paid subscription (1 point); L, literature used (1 point); I, in-app purchase (1 point); C, connectivity (1 point); A, advertisements (1 point); T, text search field (1 point); I, interdevice compatibility (1 point); O, other component (1 point); N, navigation ease (1 point); S, subjective presentation (1 point); TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

^a) Mareng Pak Mdluk” means cervical cancer in Thai language.

Table 3

App rating using specific statements of cervical cancer apps

Statement	App name																Mean score
Statement	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	Mean score
1. Cervical cancer is caused by HPV infection	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.75
2. HPV infection is contracted by sexual contact	1	1	1	1	1	0	1	0	1	0	0	0	0	1	1	1	0.63
3. HPV infection cannot be prevented by vaginal douching after intercourse	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0.00
4. HPV infection cannot be entirely prevented by using a condom	0	1	0	1	0	0	0	1	1	0	0	0	0	1	1	0	0.38
5. Smoking increases the risk of cervical cancer	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.75
6. Using combined contraceptive pills increases the risk of cervical cancer	1	0	1	1	1	0	1	1	1	0	0	1	0	1	1	1	0.69
7. Early stage of cervical cancer is asymptomatic	1	1	1	1	1	0	0	1	1	0	0	0	0	1	0	0	0.50
8. Cervical cancer can screen with a Pap smear	1	1	1	1	1	1	1	1	1	0	1	0	0	1	0	1	0.75
9. Pap smear is not solely indicated in women with vaginal discharge or bleeding	0	1	0	1	0	0	0	1	1	0	0	0	0	1	0	0	0.31
10. The HPV infection cannot be detected by conventional cervical cancer screening methods or Pap smear	0	0	1	1	0	0	0	1	1	0	1	0	0	1	0	1	0.44
11. An abnormal Pap smear result is not indicative of cervical cancer	0	0	1	1	1	0	0	1	1	0	1	0	0	1	0	1	0.50
12. A Pap smear result cannot be known immediately after the vaginal examination	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0.06
13. Most HPV infections can be healed by themselves	0	1	0	0	1	0	0	0	1	0	0	1	0	0	0	0	0.25
14. The HPV vaccine cannot provide 100% protection against HPV	1	1	1	1	1	0	1	1	1	0	0	1	0	1	1	0	0.69
15. Having multiple sexual partners increases the risk of cervical cancer	1	1	1	1	1	0	1	0	1	0	0	1	0	1	1	1	0.69
16. Sex at an early age increases the risk of cervical cancer	0	1	0	0	1	0	1	0	1	0	0	1	0	1	0	1	0.44
Total score (16 points)	8	11	11	12	11	1	8	10	14	0	3	7	0	13	7	9	7.81

1, Mareng Pak Mdluk; 2, Cervical cancer guide; 3, Cervical Cancer (Bedieman); 4, Cervical Cancer (Nature Healthy Care); 5, Cervical Cancer (Fumo); 6, MyPap; 7, Scanvix; 8, Cervical Cancer Symptoms; 9, Cervical cancer (Anastore); 10, TNM Cancer Staging Calculator; 11, BSCCP; 12, Cancer Genetics; 13, FIGO Gyn Cancer Management; 14, NCCN Guidelines^®; 15, Cervical Cancer (Personal Remedies LLC); 16, ASCCP Management Guidelines.

App, application; HPV, human papillomavirus; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

Table 4

Characteristic of cervical cancer apps

App name (developer)	Platform	Category	Language	Update dates^a)	Price (USD)	Download	Average star rating
1. Mareng Pak Mdluk^b) (Kimmydroid)	Android	Health and fitness	TH/EN	04/11/2020	0	>1,000	-
2. Cervical cancer guide (free apps for everyone)	Android	Health and fitness	EN	28/05/2020	0	>5,000	3.8
3. Cervical Cancer (Bedieman)	Android	Health and fitness	EN	11/04/2020	0	>100	-
4. Cervical Cancer (Nature Healthy Care)	Android	Entertainment	EN	20/06/2020	0	>1,000	-
5. Cervical Cancer (Fumo)	Android	Medical	EN	12/04/2020	0	>1,000	-
6. MyPap (Ovidiu Miu)	Android	Health and fitness	EN	06/02/2021	0	>50	-
7. Scanvix (Charles Ingorot)	Android	Health and fitness	EN	06/10/2019	0	>100	4.8
8. Cervical Cancer Symptoms (Revolxa Inc.)	Android	Medical	EN	01/03/2021	0	>1,000	4.8
9. Cervical cancer (Anastore)	Android	Medical	EN	22/03/2017	0	>100	4.1
10. TNM Cancer Staging Calculator (Integrated Cancer Research)	Android and iOS	Medical	EN	22/03/2018	0	>10,000	4.5
11. BSCCP (Thursday Studio)	Android and iOS	Medical	EN	22/07/2020	0	>500	-
12. Cancer Genetics (UBQO Limited)	Android and iOS	Medical	EN	18/02/2016	0	>1,000	-
13. FIGO Gyn Cancer Management (International atomic energy agency)	Android and iOS	Health and fitness	EN	09/04/2019	0	>10,000	4.7
14. NCCN Guidelines^® (NCCN)	Android and iOS	Medical	EN	24/06/2020	0	>100,000	3.0
15. Cervical Cancer (Personal Remedies LLC)	Android and iOS	Medical	EN	02/12/2019	3.49	>1	-
16. ASCCP Management Guidelines (ASCCP)	Android and iOS	Medical	EN	14/10/2020	9.99	>5,000	2.8

App, application; EN, English language; TH, Thai language; TNM, tumor, nodes, and metastases; BSCCP, The British Society for Colposcopy and Cervical Pathology; UBQO, The Specific Name of Private limited Company; FIGO, The International Federation of Gynecology and Obstetrics; NCCN, The National Comprehensive Cancer Network; LLC, Limited Liability Company; ASCCP, The American Society for Colposcopy and Cervical Pathology.

^a) Update dates are shown as (DD/MM/YYYY);

^b) Mareng Pak Mdluk” means cervical cancer in Thai language.

Table 5

Multimodal assessment tools comparison of cervical cancer apps

App name	Assessment tools			Average star rating
App name	Total MARS score (5 points)	APPLICATIONS scoring system (16 points)	Apps rating using specific statements (16 points)	Average star rating
1. Mareng Pak Mdluk^a)	2.85	9	8	0
2. Cervical cancer guide	2.58	11	11	3.8
3. Cervical Cancer (Bedieman)	2.33	9	11	0
4. Cervical Cancer (Nature Healthy Care)	2.30	7	12	0
5. Cervical Cancer (Fumo)	2.50	8	11	0
6. MyPap	3.58	7	1	0
7. Scanvix	1.95	7	8	4.8
8. Cervical Cancer Symptoms	1.96	10	10	4.8
9. Cervical cancer (Anastore)	1.52	6	14	4.1
10. TNM Cancer Staging Calculator	3.42	8	0	4.5
11. BSCCP	2.17	11	3	0
12. Cancer Genetics	1.96	6	7	0
13. FIGO Gyn Cancer Management	3.54	8	0	4.7
14. NCCN Guidelines^®	3.54	10	13	3.0
15. Cervical Cancer (Personal Remedies LLC)	1.50	8	7	0
16. ASCCP Management Guidelines	3.98	11	9	2.8
Mean±standard deviation	2.60±0.79	8.50±1.71	7.81±4.56	-

^a) Mareng Pak Mdluk” means cervical cancer in Thai language.

References

1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2021;71:209-49.

2. Medical Record and Databased Cancer Unit. Cancer in Thailand Vol X 2016 2018 [Internet] Bangkok: National Cancer Institute Medical Record and Databased, Medical Digital Division; c2021 [cited 2021 Aug 17]. Available from: https://www.nci.go.th/e_book/cit_x/index.html .

3. Bogale AL, Teklehaymanot T, Haidar Ali J, Kassie GM. Knowledge, attitude and practice of cervical cancer screening among women infected with HIV in Africa: systematic review and meta-analysis. PLoS One 2021;16:e0249960.

4. Osei EA, Appiah S, Gaogli JE, Oti-Boadi E. Knowledge on cervical cancer screening and vaccination among females at Oyibi community. BMC Womens Health 2021;21:148.

5. Weng Q, Jiang J, Haji FM, Nondo LH, Zhou H. Women’s knowledge of and attitudes toward cervical cancer and cervical cancer screening in Zanzibar, Tanzania: a cross-sectional study. BMC Cancer 2020;20:63.

6. Likitdee N, Kietpeerakool C, Chumworathayi B, Temtanakitpaisan A, Aue-Aungkul A, Nhokaew W, et al. Knowledge and attitude toward human papillomavirus infection and vaccination among Thai women: a nationwide social media survey. Asian Pac J Cancer Prev 2020;21:2895-902.

7. Kruse C, Betancourt J, Ortiz S, Valdes Luna SM, Bamrah IK, Segovia N. Barriers to the use of mobile health in improving health outcomes in developing countries: systematic review. J Med Internet Res 2019;21:e13263.

8. Salmani H, Ahmadi M, Shahrokhi N. The impact of mobile health on cancer screening: a systematic review. Cancer Inform 2020;19:1176935120954191.

9. Charbonneau DH, Hightower S, Katz A, Zhang K, Abrams J, Senft N, et al. Smartphone apps for cancer: a content analysis of the digital health marketplace. Digit Health 2020;6:2055207620905413.

10. Ana FA, Loreto MS, José LM, Pablo SM, María Pilar MJ, Myriam SA. Mobile applications in oncology: a systematic review of health science databases. Int J Med Inform 2019;133:104001.

11. Kessel KA, Vogel MME, Schmidt-Graf F, Combs SE. Mobile apps in oncology: a survey on health care professionals’ attitude toward telemedicine, mhealth, and oncological apps. J Med Internet Res 2016;18:e312.

12. Böhme C, von Osthoff MB, Frey K, Hübner J. Qualitative evaluation of mobile cancer apps with particular attention to the target group, content, and advertising. J Cancer Res Clin Oncol 2018;144:173-81.

13. Upadhyay VA, Landman AB, Hassett MJ. Landscape analysis of oncology mobile health applications. JCO Clin Cancer Inform 2021;5:579-87.

14. DeNicola N, Marko K. Connected health and mobile apps in obstetrics and gynecology. Obstet Gynecol Clin North Am 2020;47:317-31.

15. Brouard B, Bardo P, Bonnet C, Mounier N, Vignot M, Vignot S. Mobile applications in oncology: is it possible for patients and healthcare professionals to easily identify relevant tools? Ann Med 2016;48:509-15.

16. Farag S, Fields J, Pereira E, Chyjek K, Chen KT. Identification and rating of gynecologic oncology applications using the APPLICATIONS scoring system. Telemed J E Health 2016;22:1001-7.

17. US Food and Drug Administration. Policy for device software functions and mobile medical applications: guidance for industry and food and drug administration staff [Internet] Washington, D.C: U.S. Food and Drug Administration; c2019 [cited 2021 Aug 17]. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/policy-device-software-functions-and-mobile-medical-applications .

18. Shin MB, Liu G, Mugo N, Garcia PJ, Rao DW, Bayer CJ, et al. A framework for cervical cancer elimination in low-and-middle-income countries: a scoping review and roadmap for interventions and research priorities. Front Public Health 2021;9:670032.

19. Bhochhibhoya S, Dobbs PD, Maness SB. Interventions using mHealth strategies to improve screening rates of cervical cancer: a scoping review. Prev Med 2021;143:106387.

20. Lee YH, Huang LH, Chen SH, Shao JH, Lai CH, Yang NP. Effects of mobile application program (App)-assisted health education on preventive behaviors and cancer literacy among women with cervical intraepithelial neoplasia. Int J Environ Res Public Health 2021;18:11603.

21. Munoz-Zuluaga CA, Gallo-Pérez JD, Pérez-Bustos A, Orozco-Urdaneta M, Druffel K, Cordoba-Astudillo LP, et al. Mobile applications: breaking barriers to early breast and cervical cancer detection in underserved communities. JCO Oncol Pract 2021;17:e323-35.

22. Amor-García MÁ, Collado-Borrell R, Escudero-Vilaplana V, Melgarejo-Ortuño A, Herranz-Alonso A, Arranz Arija JÁ, et al. Assessing apps for patients with genitourinary tumors using the mobile application rating scale (MARS): systematic search in app stores and content analysis. JMIR Mhealth Uhealth 2020;8:e17609.

23. Jupp JCY, Sultani H, Cooper CA, Peterson KA, Truong TH. Evaluation of mobile phone applications to support medication adherence and symptom management in oncology patients. Pediatr Blood Cancer 2018;65:e27278.

24. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3:e27.

25. Farag S, Chyjek K, Chen KT. Identification of iPhone and iPad applications for obstetrics and gynecology providers. Obstet Gynecol 2014;124:941-5.

26. Sedhom R, McShea MJ, Cohen AB, Webster JA, Mathews SC. Mobile app validation: a digital health scorecard approach. NPJ Digit Med 2021;4:111.

27. Hongsanun W, Insuk S. Quality assessment criteria for mobile health apps: a systematic review. Walailak J Sci & Tech 2020;17:745-59.

28. StatCounter Global Stats. Mobile operating system market share worldwide-August 2021 [Internet] Dublin: Statcounter; c2021 [cited 2021 Aug 17]. Available from: http://gs.ststcounter.com/os-market-share/mobile/world-wide .

29. Stoyanov SR. MARS training video [Internet]. Stoyan Stoyanov c2016 [cited 2021 Aug 17]. Available from: https://www.youtube.com/watch?v=25vBwJQIOcE .

30. Kietpeerakool C, Phianmongkhol Y, Jitvatcharanun K, Siriratwatakul U, Srisomboon J. Knowledge, awareness, and attitudes of female sex workers toward HPV infection, cervical cancer, and cervical smears in Thailand. Int J Gynaecol Obstet 2009;107:216-9.

31. Lu DJ, Girgis M, David JM, Chung EM, Atkins KM, Kamrava M. Evaluation of mobile health applications to track patient-reported outcomes for oncology patients: a systematic review. Adv Radiat Oncol 2020;6:100576.

32. Ghalavandi S, Heidarnia A, Zarei F, Beiranvand R. Knowledge, attitude, practice, and self-efficacy of women regarding cervical cancer screening. Obstet Gynecol Sci 2021;64:216-25.