THE UTILIZATION OF SYSTEM USABILITY SCALE IN LEARNING MANAGEMENT SYSTEMS : A CASE STUDY OF JEDDAH COMMUNITY COLLEGE

Learning Management Systems (LMS) have been widely adopted in many higher educational institutions to support teaching and learning activities. The study was conducted with the aim of investigating the usability of the learning management system (Blackboard) at Jeddah Community College (JCC) from the students' perception, using a System Usability Scale (SUS) survey. The study examined the association between SUS scores and different variables: age, prior experience with LMS, ICT skills, attitude towards the Internet and LMS usage frequency. All participants were students at JCC. The online survey was sent by email to students who were enrolled in Information Technology (IT) courses during the 2016 summer semester. 50 responses were received and used for data analysis. The findings indicate that the overall mean of SUS score is 69.3 (SD 11.31), indicating that the current LMS has some usability issues. It was also found that the students’ LMS usage frequency is positively correlated with their SUS score. The findings can be used to consider the effectiveness of LMSs.


INTRODUCTION
In the last few decades, the development of Information and Communication Technologies (ICT) has introduced many important benefits in various areas.In higher educational institutions, ICT has transformed the manner in which university courses are delivered.The integration between ICT and education has produced various advantages.For example, ICT can enhance the quality of education, increase learning flexibility, eliminate the limitations of time and place and provide teachers and students with more control.The reason for the adoption of advanced ICT is usually a wish to improve teaching and learning effectiveness.The evolution of ICT in the field of education has produced new concepts such as e-learning, blended learning, virtual classes, web-based classes, online learning, and LMS [1].
LMS systems have been widely adopted in many higher educational institutions worldwide.They can be used for fully online learning and blended learning alike.Abazi-Bexheti, Kadriu and Ahmedi [2] reported that the move to online learning of a higher educational institution usually involves the adoption of an LMS [2].Hussein in [3] has defined LMS as software intended to manage educational processes and activities.It is an application developed with the particular goal of assisting teachers in conveying learning and knowledge to students [4].LMS provide educational institutions with the capability to share, store and manage the learning materials and content.The system is composed of many well integrated features for the service of teachers and students to meet their teaching and learning objectives.LMS can encompass the provision of course registration, upload and download learning materials, synchronous and asynchronous communication between students and teachers, assignment submission, exams and student performance assessment.The objective of an LMS is to encourage course management and collaboration among teachers and students through the utilization of Information and Communication Technologies (ICT) [5].The LMS is deployed to meet higher educational institutions' requirements for providing fully online and blended learning.
The structure of the paper is organized as follows: First, the motivation for the study is presented, followed by a brief description of the study context.After explaining the ideas behind the SUS, the methodology is detailed.The study findings are presented prior to the discussion and conclusion sections.

STUDY MOTIVATION
During the last 2 decades, many higher educational institutions have adopted ICT in order to enhance education processes, activities and outcomes.In response, to be in accordance with the advancement in ICT and education, JCC adopted an LMS in 2014 (Blackboard) and insisted that faculty members should utilize the system.Evaluating and monitoring the usability of an e-learning system is considered to be an important task in order to ensure the effectiveness of the system [6].Moreover, many studies [7,8,6] have concluded that users of such a system often experience many usability challenges.

JEDDAH COMMUNITY COLLEGE
Jeddah Community College (JCC) is a 3-year higher education institute in Jeddah, Saudi Arabia.JCC was established in 2002 as one of King Abdulaziz University (KAU) colleges with the aim of serving the community of Jeddah city.The college provides students with professional programs and classes to meet the needs of the business market, such as Computer Networks, Accounting, Graphic Design and Insurance.According to the Saudi Arabian Ministry of Education, 932 students are enrolled at JCC [9].The college is adopting ICT in teaching as well as administrative transactions.However, LMS utilization at JCC is still in its infancy.

SYSTEM USABILITY SCALE
In 1996, the SUS was introduced by John Brooke with the object of developing a quick, reliable and inexpensive method for evaluating the usability of technologies [10].The SUS is a survey that consists of only 10 statements (Table 1), and participants agree or disagree with the statements based on 5-point Likert scale.A SUS score can be within the range of 0 and 100 for each participant.The score indicates the usability level of the technology from the participants' point of view without providing any explanation for the participants' perception [11].
Although many surveys were developed for evaluating the usability of systems, SUS has some distinguishing features.It is flexible enough to measure the usability of various systems, technologies, hardware and software [12].The SUS is simple and inexpensive compared to many other methods of system usability evaluation.Moreover, the questionnaire can be completed quickly because it is short and contains only 10 statements.The SUS score is between 0 and 100, which makes it understandable by experts and non-experts alike.Finally, many studies [6,12] have concluded that SUS is a valid and reliable tool even for a small sample size.However, Orfanou et al. reported that in spite of its advantages, SUS has rarely been used in evaluating LMS [6].
Table 1.SUS statements SUS Statements 1 I think that I would like to use this system frequently.2 I found the system unnecessarily complex.3 I thought the system was easy to use. 4 I think that I would need the support of a technical person to be able to use this system.5 I found the various functions in this system were well integrated.6 I thought there was too much inconsistency in this system.7 I would imagine that most people would learn to use this system very quickly.8 I found the system very cumbersome to use. 9 I felt very confident using the system.10 I needed to learn a lot of things before I could get going with this system.

Acceptable SUS Score
Although a SUS score is quantitative and understandable by non-experts, SUS does not indicate the acceptable usability score.Therefore, Bangor et al. in [12] produced a scale for an acceptable SUS score.A SUS score between 85 and 100 indicates that the system is highly usable.The usability of the system is excellent when the SUS score is between 70 to about 85.With a SUS score from 50 to about 70, the acceptability is good however the system users experience usability issues.Finally, the usability of the system is not acceptable when the SUS score below 50.This scale of acceptable SUS scores is used for the purpose of the study.

SUS Variables
This study investigates the association between SUS score and different variables: age, prior experience with LMS, ICT skills, attitude towards the Internet and LMS usage frequency.Orfanou et al. concluded that there is an association between SUS score and Internet self-competence, LMS experience, perception toward the Internet and LMS utilization [6].In other contexts, it was concluded that the participants' age is significantly correlated with SUS score [12], and users' prior experience is also a factor [13].

SUS Validity and Reliability
The SUS is considered as one of the most reliable surveys in measuring the usability of various technologies.Tullis and Stetson in [14] evaluated the usability of www.yahoo.comand www.kiplinger.combased on 5 dissimilar surveys.SUS was the most reliable among the 5 surveys.Tullis and Stetson concluded that the SUS provides reliable results in 90-100% of instances when the sample size is between 12 to 14 and 75-80% of instances when the sample size is only 10. Orfanou, Tselios and Katsanos have examined the reliability of SUS in evaluating the usability of LMS and concluded that SUS provides reliable results even with a sample size between 6 and 14 [6].

METHODOLOGY
This methodology section mainly focuses on the method used for data collection.The survey instrument, pilot study and participants will be discussed in this section.Data analysis will also be described.

Data Collection Method
The SUS survey is normally quantitative in nature; an online survey was therefore used for data collection.Heiervang and Goodman have concluded that the online survey is faster and less expensive than the traditional methods [15].Moreover, the probability of mistakes and inaccuracy is diminished with online surveys as participants interact directly with the system.Google Forms was selected as the medium of the online survey due to the affordability and ease of use for researchers and participants alike.

Instrument
The instrument of the study comprises 3 sections.The first section describes the study objective, context and confidentiality of participation.The second section includes the participants' demographic information and consists of 6 questions: age, gender, prior experience with LMS, ICT skills, attitude towards Internet and LMS usage frequency (Table 2).The final section includes the SUS 10 statements and can be answered using the 5-point Likert scale, where 1 means strongly disagree and 5 means strongly agree.

Pilot
It was necessary to conduct a pilot prior to the main study, to remove unforeseen problems in the method.All participants of this study were students at JCC and enrolled in Information Technology (IT) courses.The survey was sent by email to 5 students selected randomly during the 2016 summer semester.All 5 students were male as there were no female students enrolled during the 2016 summer semester.Having completed the survey, one of the participants' enquiries was about the meaning of the term "cumbersome" in statement 8.To make the statement more understandable, the term was substituted in the study by another synonym "awkward".Interestingly, the same enquiry was raised by the participants of another study where the use of the word "awkward" was justified [12].Furthermore, the term "system" was replaced with "Blackboard".

Participants
All participants of this study were students at JCC and enrolled in IT courses.The link to the online survey was sent by email to 81 students selected randomly during the 2016 summer semester.The survey was available for 1 month for the participants.All 81 students were male students as there were no female students enrolled during the 2016 summer semester.50 responses were received and used for data analysis, which corresponds to a response rate of 61.7%.

Data Analysis
After the completion of the data collection stage, the responses were downloaded from Google Forms, updated in Microsoft Excel 2016 and transformed to SPSS Statistics 20 for further statistical tests.Column and pie charts were used to represent the demographic information and SUS score.A normality test was performed using SPSS to ensure whether the data was normally distributed.To test the statistical significance differences, Kruskal-Wallis H test, Mann-Whitney U test and independent-samples t-test were utilized.The internal consistency of the SUS survey was measured using Cronbach's alpha.Furthermore, Pearson correlation test was employed to examine the association between SUS score and the variables of age, ICT skills and LMS usage frequency.

Demographic Information
All the participants were male students enrolled in IT courses at JCC.The majority (88%) of the students were between 20 and 25 years old.Only 12% of students were either younger than 20 or older than 25.In Fig. 1, 51% of the students asserted that they did not have prior experience with a LMS before joining JCC.This is not surprising as the utilization of LMS is not widespread in Saudi Arabia [16].All participants reported a positive attitude towards the Internet as a good learning tool.However, Fig. 2 depicts that only 30% of participants have advanced ICT skills and 70% of the participants have basic ICT skills.The findings are surprising since the participants are enrolled in IT courses.

Fig. 2. Distribution of ICT skills
Fig. 3 shows frequency of use which varied amongst the participants.The majority of the participants were using the system at a rate appropriate to their level and intensity of study; however, the system was underused by 12% of the participants.
Fig. 3. Distribution of LMS usage frequency The findings demonstrate that the overall mean of the SUS score was 69.3 and the standard deviation was 11.31.According to the scale of acceptable SUS score, the usability level of the system is good while recognizing some usability issues exist.Fig. 4 confirms that 54% of students rated the usability level of the LMS as good.It is noteworthy that specific usability issues cannot be identified by SUS as SUS is not a diagnostic tool.

Overall SUS Score
Fig. 4. Acceptable SUS score scale

SUS Validity and Reliability
Although many studies have concluded that SUS is highly reliable, a reliability assessment was performed for this study.As reliability is widely measured using Cronbach's alpha [6], this method was employed to examine the internal consistency of SUS.Sekaran and Bougie reported that reliability is acceptable if Cronbach's alpha value is greater than 0.7 [17].The findings indicate that SUS is reliable (10 items, α = 0.707).

SUS Score by Age
The mean SUS score by age is depicted in Fig. 5. Students between 20 and 25 years old rated the usability of the system higher compared to the students younger than 20 or older than 25 years old.

54% 34%
12% Good Excellent High Fig. 5. SUS Score mean by Age A test was performed using SPSS to ensure the data normality.According to the Shapiro-Wilk test [18], normal Q-Q plots, box plots and histograms, the data was found to be normally distributed.An independent samples t-test was used to demonstrate that there was no significant difference in SUS score between age groups (t(48) = 1.59, p = .119).This finding is inconsistent with the results of [12] but consistent with the results of [6].From the Pearson correlation test, there is no correlation between age and SUS score, r=-.223, n=50, p=.119.

SUS Score by Prior Experience with LMS
Although the difference is slightly noticeable, Fig. 6 shows that students who have prior experience with an LMS before joining JCC rated the usability of the system lower than students with no prior experience with an LMS.This finding is inconsistent with the results of [6], where students with prior experience rated the usability 2.12% higher.
Fig. 6.SUS Score mean by Experience with LMS A normality test was performed using SPSS to ensure the data normality.When the z-value for Kurtosis or Skewness is not within the range of -1.96 and +1.96, the data is not normally distributed [19].As the zvalue of inexperienced student Skewness is 2.17, the data is significantly different from a normal distribution.Therefore, a non-parametric test was employed [6].The Mann-Whitney U test demonstrated that there was no statistical significance difference in SUS score between experienced and inexperienced students (U = 279, p = .855).This finding differs from the results of [6].A normality test was performed using SPSS to ensure the data normality.According to the Shapiro-Wilk test [18], normal Q-Q plots, box plots and histograms, the data was found to be normally distributed.An independent samples t-test was used to demonstrate that there was no significant difference in SUS score between ICT skills groups (t(48) = -.305,p = .762).From the Pearson correlation test, there is no correlation between ICT skills and SUS score, r=.044, n=50, p=.762.Therefore, the level of ICT skills is not associated with SUS score.

SUS Score by LMS Usage Frequency
The mean of SUS score by LMS usage frequency is depicted in Fig. 8, which shows that SUS score increased by usage frequency.

Fig. 8. SUS Score mean by LMS Usage Frequency
A normality test was performed using SPSS to ensure the data normality.According to the Shapiro-Wilk test, normal Q-Q plots, box plots and histograms, the data was found to be normally distributed.A Kruskal-Wallis H test demonstrated that there was a significant difference in SUS score between LMS usage frequency groups (X 2 (2) = 7.376, p = .025).From the Pearson correlation test, there is a moderate correlation between LMS usage frequency and SUS score, r=.376, n=50, p=.007.Therefore, LMS usage frequency is positively associated with SUS score.

DISCUSSION AND CONCLUSION
In general, the study investigated the usability of an LMS (Blackboard) at JCC from the students' perspective using a SUS survey.The scale of acceptable SUS scores [12] was used for the purpose of the study.The statistically significant differences and correlation between SUS score and other variables (age, prior experience with LMS, ICT skills, attitude towards Internet and LMS usage frequency) were assessed.
The findings indicate that the overall mean of SUS score is 69.3 (SD 11.31), which indicates that the users experiences some usability issues.All participants had a positive attitude towards the Internet as a good learning tool, which makes Saudi Arabia a fertile soil for e-learning adoption and utilization in the near future.In addition, the students' LMS usage frequency was found to be positively associated with the SUS score, and there is a statistically significant difference in SUS score between LMS usage frequency groups.However, there are no significant differences or correlation between SUS score and prior experience with LMS, ICT skills and age.The study has shown that there is a need for more in-depth exploration of the nature of the usability concerns that the SUS scores indicated.
This study is not free of limitations.Female students were not involved in the study because there were no female students enrolled during the 2016 summer semester.For this reason, another study is planned to be conducted to examine the usability from the perception of both genders.In addition, the participants were IT students at JCC.The scope of the study can be expanded to include students from different colleges or universities and various fields of study.Finally, this study investigated only the students' perception.At a later time.teachers and administrators can be added to the scope of the study.

Fig. 1 .
Fig. 1.Distribution of prior experience with LMS

Fig. 7 Fig. 7 .
Fig.7depicts the mean of SUS score by ICT skills.Although students with advanced ICT skills rated the usability of the system higher compared to other students, the difference is only slightly noticeable.

Table 2 .
Demographic Information

Table 3
summarizes the participants' responses frequency and percentage, mean and standard deviation of each statement.However, the overall mean of the positive statements is 3.94, which locates the responses between neutral and agree.Statement 7 has the highest mean (4.12), and Statement 1 has the lowest mean (3.80) among the positive statements.On the other hand, the overall mean of the negative statements is 2.40, which locates the responses between disagree and neutral.Statement 10 has the highest mean(3.18),and Statement 2 has the lowest mean (2.06) among the negative statements.