The USSD Mobile Phone App
The final paper-based version of the adolescent reproductive health content (Additional file 1: Appendix A) was programmed into the web-based interface provided by Echomobile® for the USSD platform.
To open the app, users dialed a 7-digit code on a mobile phone, which then prompted them to input a pre-assigned PIN linked to their cellphone number. When the users were authenticated, they selected their gender and age on subsequent screens before selecting a sexual reproductive health topic of interest. The users then interacted with screen-by-screen content guided by their input. A selection of the mobile app interactive screens is shown on Fig. 2.
Connecting to the USSD app
Adolescent users could select sexual reproductive health topics of interest and access information. Figure 3 shows how a user accessed content on sexual relationships.
Accessing specific sexual reproductive health information in the USSD app
Alpha Testing
For the alpha testing, participants were required to dial a 7-digit USSD code on either a feature phone or smartphone., The users were then taken through authentication and selection of gender and age. Table 1 shows the demographic characteristics of the adolescents who were successfully followed up and had used the app at least once during the 1-month period of alpha usability testing. The median age of participants was 15 years for the under 18 and 18 years for the 18 and above group. All attended secondary school. Of the 38 adolescents enrolled, 12 were successfully followed up, only 9 had used the app at least once during the 1-month period. Only survey questions applicable to the USSD app were analyzed. Questions on app customization, sharing, app description on Playstore, buttons and icons were excluded.
Participants feedback on engagement, functionality, aesthetics, and quality of information provided by the mobile app is shown in Table 2. The MARS was scored 1 to 5 with 1 being the lowest score and 5 the highest. The highest scores were attained on engagement of the app with a mean score of 4.4, and functionality with a mean score of 4.3. The MARS consisted of 15 items and the value for Cronbach’s Alpha for the survey was α = 0.83.
Under the aesthetics, information on layout, the “arrangement and size of buttons / icons” and graphics “the quality / resolution of graphics used for buttons / icons / menus / content” were excluded from the final analysis. The research team noted that these features were not applicable to the USSD app, the app contains no buttons, icons or graphics. Since no issues were identified during the alpha testing of the app, nothing was changed before the field testing.
Field Usability Testing
Once the alpha usability test had been completed and performance of the app validated, field usability testing was carried out. Participants had to dial a 7-digit USSD code on either a feature phone or smartphone, before being taken through authentication and selection of gender and age. Once in the app, the users had a list of options to choose from based on their information needs. For the field testing, 146 adolescents were enrolled, 113 were followed-up, and 109 provided complete study data. Among these, 62 had used the app at least once in the 3-month period. The demographic characteristics of participants who used the app during the field usability testing are shown in Table 3. This p-values checked potential statistically significant difference in demographic characteristics between participants under 18 years and above 18 years. The mobile application rating scale scores for the field usability testing are shown in Table 4. Only survey questions applicable to a USSD app were analyzed. Questions on app customization, sharing, app description on Playstore, buttons and icons were excluded.
During the field usability testing, engagement of the app attained a mean score of 4.3 (0.44). The functionality mean score was 4.6 (0.38), with navigation and gestural design ratings within the functionality score attaining a mean score of 4.8 (0.43) and 4.8 (0.35) respectively. The overall mean score for information was 4.4 (0.31), with quantity of information attaining 4.6 (1.11) and quality of information 4.5 (0.71). The value for Cronbach’s Alpha for the field testing was α = 0.54. It is important to note that, 109 adolescents were successfully followed up after the 3-month period. However, only 62 had used the USSD app at least once. The usability testing interview was only administered to adolescent participants who had used the app at least once during the 3-month period. Table 5 shows participants feedback on the app’s characteristics. Only options selected by users are included in this table.
The adolescents found the app entertaining, with 43.6% (27) of the users indicating the app was fun to use. Most of the users, 70.9% (44), found the app very interesting. The adolescents deemed the app content to be appropriately directed, with 54.8% (34) users indicating it was perfectly targeted. Only 3.2% (2) of the users felt the content was not well targeted or inappropriate.
On the accuracy of app features, 72.5% (45) of the adolescents felt the features were perfect and did not experience any bugs. The adolescents were able to learn how to use the app swiftly, with 72.6% (45) finding this easy. Regarding the app’s interaction, 85.5% (53) of the adolescents found the content consistent and intuitive across all screens.
Over 90% of the adolescents found the content provided in the app relevant to their sexual reproductive health needs. Gender differences were not significant. The content was comprehensive according to 87.1% (54) of the adolescents. Notably, at least one in every two participants, 54.8% (34), indicated they would recommend the app to other adolescents. The app was rated highly, with 72.6% (45) of users describing it as the ‘best app’ for providing adolescent sexual reproductive health information.
Although 45.2% (28) of users ‘strongly agreed’ the app could increase awareness of sexual reproductive health information, there were important gender differences within this category. While 66.7% (20) of male participants ‘agreed’ the app could increase awareness, less than half the female users, 43.7% (14), ‘agreed’. The majority of female participants, 56.3% (18), ‘strongly agreed’ the app could increase awareness. On increasing knowledge, 59.7% (37) of users ‘agreed’ the app was likely to increase knowledge on sexual reproductive health.
Users also felt that the app could change attitudes toward adolescent sexual reproductive health, with 69.4% (43) agreeing that a change in attitude would improve service provision and uptake. The app was envisioned to potentially increasing the uptake of interventions providing adolescent sexual reproductive health by 81.0% (44) of participants. On whether the app would encourage help-seeking behavior by adolescents on reproductive health issues, 66.1% (41) of participants agreed. A reduction in adolescent sexual reproductive health problems through use of the app was predicted by 66.1% (41) of participants.
A number of user experience evaluation components attained substantially different scores from male and female users. On the overall rating of the app, 72.6% (45) rated the app as the ‘best app’: of these, 84.4% (27) were female users compared to 60.0% (18) male, with a p-value = 0.011 . On the USSD app increasing awareness of addressing adolescent sexual reproductive health, 56.3% (18) of the female users ‘strongly agreed’ compared to 33.3% (10) male users. On the likelihood of the app increasing knowledge on sexual reproductive health, 46.9% (15) of female users ‘strongly agreed’, but only 30.0% (9) of male users.