Quality assurance (QA) in higher education (HE) is emerging as the higher education institutions (HEIs) find it difficult to monitor their levels of quality of services. Governments impose national QA standards in order to achieve the minimum level of quality of services provided by these institutions. In the Kingdom of Saudi Arabia (KSA), Education and Training Evaluation Commission (ETEC) imposes, through its National Centre for Academic Accreditation and Assessment (NCAAA), the QA standards that all HEIs in the KSA are required to comply with for accreditation [1]. To measure compliance with these standards, in 2018, NCAAA developed 23 key performance indicators (KPIs). In 2022, NCAAA introduced a revised version of KPIs. This revised version comprises a total of 17 KPIs, and all HEIs are mandated to monitor and report on compliance. This evaluation process of compliance with these KPIs should take place during the forthcoming accreditation cycle for HEIs [2].

In addition, the increasing amount of data generated from traditional sources and through social media plays negative influence on business intelligence (BI). There is a desire to build low-cost data platforms that will be able to handle this increasing amount of data. This is sometimes known as the big data challenge [3,4].

In this paper, different BI architectures are presented. These BI architectures encompass different BI technologies that HEIs may adopt, taking into consideration their capabilities and specific requirements of HEIs. A proposed architecture is also presented to show how the big data challenge can be addressed to deal with huge amounts of data that the QA system might require particularly if the data is being linked to social media. The recent trend of measuring students’ satisfaction is through students’ comments posted on social media.

As discussed earlier, the NCAAA standards in the KSA were developed in 2018, and a revised version was released in 2022. Table 1 compares KPIs released in 2018 and 2022 and shows the related standards that are aimed to be measured. The comparison indicates that the revised KPIs include the same KPIs as that in the 2018 version but 5 KPIs have been omitted, while the revised QA standards released in 2022 remain the same (8 QA standards). The omission of these institutional KPIs was not expected as there are no KPIs specified now for measuring the standard 8 (Community Partnership). This reduction in KPIs indicates that the QA systems in HEIs in the KSA are now more mature and also allows the institutions to include additional specific KPIs if they wish to.

The NCAAA standards released in 2018 have been benchmarked against international standards [5]. In this study, the NCAAA standards released in 2022 are benchmarked against several international standards including Academic Quality Improvement Programme (AQIP) in the USA, Southern Association of Colleges and Schools (SACS) accreditation standards in the USA, Scottish Qualifications Authority (SQA) in Scotland, European Network for Quality Assurance (ENQA) in HE accreditation standards, Tertiary Education Quality and Standards Agency (TEQSA) in Australia, New Zealand Qualifications Authority (NZQA) standards, and Quality Assurance Agency (QAA) for HE standards in the UK. The comparison is presented in Table 2, which demonstrates that each of the NCAAA standards released in 2022 can be mapped to a similar international standard.

A holistic framework discussed in this paper was used to develop the prototype dashboard which supports the NCAAA standards released in both 2018 and 2022. It is also noted that some institutions treat the NCAAA KPIs as a minimum requirement and may use additional institution specific KPIs. For example, King Saud University adopted a total of 56 KPIs to measure QA performance [6].

A holistic framework for monitoring quality in HEIs using BI dashboards in the KSA (HF-HEQ-BI) was introduced previously [7]. The holistic framework and the prototype dashboard developed from the framework must therefore also be able to support additional KPIs if required by the institution. There was a need to identify the requirements to design the BI system, which assist decision-makers to monitor compliance with QA standards in HEIs. It was also necessary to identify the factors that HEIs must monitor and consider while developing the BI system for QA monitoring in the KSA. An extensive literature review identified 52 studies that discussed QA in HE using BI systems [8], indicating that there are 45 factors identified. However, none of these studies indicated the role of social media in QA monitoring using BI dashboards. A hybrid framework has been developed through the systematic literature review conducted using five main pillars (technology-organisation-environment-business-social) based on the information system strategic triangle, technology-organisation-environment (TOE) framework, and human, organisation and technology-fit to provide the theoretical underpinning background [9–12]. Alharbi et al. [9] and Maroufkhani et al. [10] used TOE, diffusion of innovation, and resource-based view theoretical frameworks for small and medium sized enterprises (SMEs). Alaboudi et al. [11] developed a framework to support decision making for healthcare organisations for telemedicine networks in the KSA, and Tashkandi and Al-Jabri [12] used TOE to investigate the adoption of cloud computing in HE.

A qualitative analysis was carried out by interviewing a panel of experts to confirm the thematic analysis of factors [13]. A total of ten experts were interviewed through semi-structured convergent interviews. The responses obtained were analysed through the computer assisted qualitative data analysis software (CAQDAS), which identified 42 factors. The quantitative analysis was then conducted to validate the second version of the framework. The sample size was calculated and confirmed from literature [14–17]. Snowball sampling was used to administer the online questionnaire [18,19]. IBM SPSS was used to conduct the statistical analysis. Cronbach’s alpha was calculated to determine the internal consistency of factors under each pillar. In addition, the confirmatory factor analysis (CFA) was conducted to calculate the relationship between the factors and determine if non-significant factor loadings existed [17,20,21]. T-test was conducted to determine whether significant differences in responses existed among public and private institutions [20,22]. Therefore, the 42 factors were confirmed to monitor QA in HEIs using BI dashboards with the triangulation approach to provide the HF-HEQ-BI framework which has been explained previously [8,13]. The factors outlined are shown in Table 3.

HEIs are collecting increasing amounts of data related to monitoring quality of services provided [23,24]. Social media has become a crucial part of QA monitoring in HEIs as numerous public opinions are expressed in social media channels [25–27]. This huge amount of data is making increasing pressure on the BI systems used for monitoring compliance with QA standards. In the BI architecture proposed by Sorour et al. [28], the BI system was fed by different data sources including social media data. Dealing with this huge amount of data requires a BI architecture that takes into consideration the emerging technologies such as service-oriented business intelligence (SoBI) and self-service business intelligence (SSBI).

Big data refers to the use of big data for system inputs. The big data technology has been used by Li et al. [29] to perform data collection, cleaning, processing, and storage of teaching information in colleges and universities through building a quality monitoring platform.

In HEIs, the type and volume of data gathered related to the QA requirements are increasing significantly. HEIs are struggling in managing data that feeds QA systems due to several challenges including:

1) HEIs are increasing in their sizes (Qassim University in the KSA, for example, has expanded to 38 colleges with approximately 58267 students ranging from diplomas to Ph.D. candidates in which 92% are undergraduates) [30] and are operating in several satellite locations;

2) As the programmes being provided by HEIs are expanding each academic year, the QA related data is also increasing;

3) HEIs are concerned with the continuous improvement which requires a cross-year analysis that encompasses huge amounts of data.

Besides, HEIs are needing to deal with huge amounts of data from social media comments by stakeholders. As the level of quality of services provided by HEIs can be partially determined by the public opinions expressed in social media [25,26]. Dealing with these huge amounts of data incorporates a challenge for HEIs in terms of identifying patterns in the expressed opinions in social media to determine what is useful and what is not for QA monitoring. Huge investments in the infrastructure of HEIs may be required as it might require multiple servers to handle the data [31]. As HEIs differ in their sizes and available resources, it might be challenging to satisfy this requirement to handle the big data challenge. In this paper, different BI architectures that could handle the big data challenge are discussed. Some of these architectures are designed to suit SMEs as well as large enterprises outlined in the following section. Therefore, each HEI might choose the appropriate BI architecture based on its own size and available resource.

This section outlines different BI architectures in HEIs to address the big data challenge. These architectures are aimed to show different configurations of BI systems in HEIs, thus helping HEIs choose the most suitable architecture according to their sizes and requirements for QA monitoring.

A data warehouse (DW) based BI architecture in HEIs has been proposed and outlined [28]. The proposed architecture suggests a traditional DW based BI architecture in HE, where the data source layer, the data movement, storage, and processing layer, and the data visualisation and reporting layer are included.

Recent studies suggested that the extract, transform, and load (ETL) process in BI systems can be handled differently by using the SoBI approach [32,33]. Fig. 1 outlines the process of SoBI through web services. Web services may be used to retrieve data directly from data sources and process it in the Extensible Markup Language (XML) format. Data is then processed using the dashboard application that will store it in DW. Because the data needs further processing and then being transformed into appropriate information for the marketing analysis using data mining techniques. A web service approach is adopted rather than the traditional ETL process, because the web service approach allows retrieving the latest data from the data sources directly. This process may process data in an easier and faster manner in comparison with the traditional BI infrastructure in the circumstances where the organisation cannot afford the costs and expertise associated with developing the DW or ETL process [33]. Web services can bridge the information from different sources and overcome the difference between technologies used in data sources (e.g., some data sources may use Oracle databases while other may use MySQL). Web services also have the ability to conquer the challenge of different data sources as XML is used to encode the data.

According to Bentley [34], embedded analytics is designed, which makes BI more accessible for all kinds of users. It allows more real-time autonomy and self-service of data visualisation and customisation. Prem considered this kind of approach as SSBI, which allows business users to access corporate information without the involvement of the information technology (IT) team [35].

Many tools can be used to obtain the benefits of data analytics without building complex BI structures or DW [36–38]. Among these tools, Tableau, Microsoft Power BI, QlikView, and Birst [36] are available from specialised vendors. SSBI involves gathering data from data sources and inserting it directly into the tool. The tool enables the user to further process data or visualise it through BI dashboards as well as producing reports [34,36,39]. This approach is presented in Fig. 2, which identifies how organisations may benefit from commercial tools for presenting data analytics. Large firms may adopt the conventional BI architecture proposed by Sorour et al. in 2019 [28] as they have the financial capabilities for implementation, but SMEs may use the SSBI approach because it is more affordable for them [38,40]. Conventional BI implementation also requires high level of knowledge and expertise that may not be available in SMEs [40]. This is of particular importance in the KSA as private universities and some of the public universities in the KSA are considered SMEs. Technologies, such as cloud computing and, in particular, software as a service (SaaS), provide smaller organisations with access to BI systems with no complex infrastructures which are required to be owned or implemented.

As discussed above, different methods can be used to realize BI systems, whose requirements are determined by the needs and capabilities of each organisation. Thus, organisations that can afford the implementation costs of the complex BI system may choose a DW based approach. While for the organisations that cannot afford the implementation costs and abilities of DW or cannot handle the complex ETL process, the web service or self-service tools are beneficial to visualise data into the dashboards. Due to the fact that HEIs in the KSA differ in sizes and technical capabilities, an alternative BI architecture is proposed in this paper, as shown in Fig. 3. Thus, each organisation might select the most suitable BI solution on the basis of the organisation’s needs and requirements.

As shown in Fig. 3, there are three paths in the proposed alternative BI architecture: Grey path, red path, and purple path. The grey path indicates the implementation that does not require any traditional DW, while the red path shows a traditional ETL and DW implementation for organisations that can afford the associated costs and for which this approach is appropriate. The purple path shows the SSBI approach, which is an alternative for SMEs or users who are not capable to afford complex BI architectures that include DW. The grey-dashed line in Fig. 3 indicates an optional route for HEIs to implement BI systems. The proposed BI architecture takes into consideration different data sources. These data sources can deal with huge amounts of data, especially that coming from social media.

The proposed HF-HEQ-BI architecture has been validated by Sorour et al. in 2019 [28]. An HF-HEQ-BI framework utilisation tool (FUT) has also been proposed by them in 2021 [7] to demonstrate how the HF-HEQ-BI framework can be used to develop dashboards for monitoring quality in HE in the KSA. A prototype dashboard, based on the utilisation tool, is presented in this paper to illustrate the way in which the HF-HEQ-BI framework can be applied in practice. A screen shot of part of the utilisation tool is presented in Fig. 4. This indicates how the measurement and information determination task can be mapped to the graphical presentation facilities in Microsoft Power BI to provide appropriate visualisation. HF-HEQ-BI FUT assists in identifying necessary visualisation for the dashboard. These visualisation icons enable decision-makers to monitor the progress regarding compliance with NCAAA standards, institutional specific KPIs, and social media analytics. For example, an institution may choose a ‘gauge’ visual to represent the number of publications while another may tend to represent this measurement through the line chart to view the annual trend of publications. The prototype dashboard was developed using Microsoft Power BI, and the visualisation to be presented in the dashboard screens was determined through the utilisation of HF-HEQ-BI FUT.

For the purpose of developing the dashboard, the Microsoft Power BI Desktop is adopted as it is intuitive to design and free to Microsoft users. For example, Noonpakdee et al. [41] used the Microsoft Power BI Desktop to develop a dashboard template for SMEs, and Sluijter and Otten [42] used it to develop personalised student dashboards.

The development of the BI dashboard includes identifying the requirements, producing the design solution, and evaluating the designed dashboard. In this section, the dashboard development phases are outlined in more detail. Fig. 5 illustrates how users can apply the HF-HEQ-BI framework to develop the dashboard for monitoring quality in HE. It is obvious that the SSBI tool obtains the inputs from users responsible for gathering data related to evaluating the QA performance. The developed prototype dashboards allow the top management to monitor QA activities and determine the areas that should be pay more attention to. The end users of the dashboard should only have restricted access to the data specific to their roles in the organisation to maintain data integrity. For example, if a department head is using the dashboard, only the data related to the specific department will be displayed in the dashboard.

During the development process, an approach based on communities of practice (COP) was used to solicit constructive comments about the dashboard design from experts with experience and management roles in QA in the KSA. The prototype dashboard was refined through several iterations using the COP approach. COP are considered useful for socialisation as they allow researchers to obtain valuable knowledge related to their fields of expertise [43]. The dashboard was presented to a panel of experts and senior academics who served as senior managers in QA in the KSA and UK to obtain their opinions in the dashboard presentation, and the several comments suggested were used to refine the presentation.

The initial dashboard only presented the design solution produced by Microsoft Power BI, leading to limited details presented in the dashboard. NCAAA KPIs were presented in the dashboard under the five pillars of the HF-HEQ-BI framework. Following the comments received, the dashboard was further improved to add details related to QA KPIs in order to monitor all QA factors. It was decided to represent the dashboard through a web portal with a navigation bar on the top of the page. A top pane was included to provide additional information about the project and how the HF-HEQ-BI framework can be used to develop the QA monitoring dashboard. The navigation bar allows users to navigate through the dashboard, use its functionalities (drilling-down, drilling-through, filtering, etc.), and return back to the home screen to assist in navigation.

The use of web portal for publishing the dashboard facilitates management in assigning user permissions and roles. The top management of HEI would have full access to view all details on the dashboard. The college’s dean would normally have access to the details related to his/her college only.

The HF-HEQ-BI framework is used to determine the context of the dashboard. The developed prototype dashboard will be used to monitor quality in HEIs. Additionally, the designed dashboard using HF-HEQ-BI takes into consideration the mandatory requirements of NCAAA and provides social media analytics. The dashboard presented is illustrated with Twitter-based analytics, it can also be extended to other forms of social media analytics. The sentiment analysis of the text-based interactions (comments) and other social media analytics will appear in the dashboard to allow the top management to monitor them. Some HEIs may add further KPIs which can sometimes be many as 50 to monitor strategic plan alignment or financial metrics. For example, King Saud University adopted a total of 56 KPIs to measure QA performance [6]. Qassim University adopted 50 KPIs to monitor strategic management performance [44]. Majmaah University adopted 58 KPIs to monitor strategic performance in addition to the NCAAA mandatory KPIs [45]. These KPIs may be presented in the dashboard in the business pillar to provide improved visualisation of institutional performance, such as the successful application rate and diversity of students.

HF-HEQ-BI FUT has been used to determine the requirements and visualisation desired for designing the dashboard. A dashboard has been developed by using an SSBI tool. The reason for choosing an SSBI architecture is that SSBI allows more flexibility and saves time [38]. The SSBI tool also allows dashboards to be developed with no need to heavily invest in developing DW. Designing DW for this project is not practical due to the time and data constraints. As one of the most popular SSBI tools, Microsoft Power BI was selected to implement the dashboard in this study.

For the purpose of developing the prototype dashboard, the SSBI tools of Microsoft Power BI and Tableau and the SoBI tool of Cube.js were reviewed and tested. Tableau offers various features, including compatibility for installation on both the Macintosh Operating System (MacOS) and Windows. It can provide a trial for all of its functionalities for one year to all students studying in academic institutions. However, Tableau does not support the Arabic language in its interface. Node.js is SoBI which requires web hosting services to utilize its functionalities. Also, web programming skills are required to portray the visualisation to fit the web pages. As a comparison, the Microsoft Power BI Desktop provides user-friendly design interface, and it is freely accessible to Microsoft users.

The visualisation requirements have been identified through HF-HEQ-BI FUT, and a dashboard has been used for the development process. Fig. 6 shows a Hypertext Preprocessor (PHP) webpage which is the homepage of the system. The top pane of the homepage provides users with information about the dashboard and the HF-HEQ-BI framework and also includes a navigation bar to navigate to different pages.

The developed dashboard has been divided into several screens to allow decision-makers to monitor QA performance and compliance with NCAAA and QA standards by HEI. A portal was presented and used for monitoring. Hyper Text Markup Language (HTML) pages were designed to embed the dashboard pages from Power BI. Fig. 7 illustrates the first page of the dashboard, which displays the mandatory KPIs and measures the degree of compliance with the respective KPIs in HEI. Users are able to drill-down and then investigate the performance related to a specific college or department. Decision-makers in HEI will be able to monitor compliance with KPIs and identify the areas where special attention is required to achieve/maintain the national NCAAA accreditation. The dashboard is dynamic and agile, which indicates that the shown KPIs can be changed to show the 23 KPIs or the new 17 KPIs depending on the current QA system used in HEIs.

In Fig. 8, the dashboard shows the institutional specific KPIs, which are related to the factors measuring compliance with QA standards that have not been covered by the NCAAA standards. Also, the dashboard is agile and dynamic, allowing HEIs to add their own specific KPIs (for example, King Saud University has up to 56 KPIs). As QA monitoring encompasses the activities related to the aspects such as ‘locations covered by the institution’ and ‘health and safety requirements’, decision-makers will be able to monitor compliance with such factors through this screen. This information identified throughout the development process of the HF-HEQ-BI framework is essential to build the QA system in HEIs.

The dashboard of social media analytics is shown in Fig. 9, which includes the sentiment analysis of Tweets coming from Twitter. In certain private HEIs in the KSA, the top management consistently monitors Twitter feeds to stay abreast of any negative publicity, given that some institutions may receive several hundred Tweets daily.

To obtain the data related to Tweets from Twitter, the Twitter application programming interface (API) was integrated into the dashboard. API allows the user to find Tweets for specific periods of time and export the sentiment analysis data into spreadsheets, thus allowing the SSBI tool to represent the data. Some SSBI tools also allow the users to connect to other third-party sentiment analysis providers such as Hootsuite and Brand24.

HEI may be required to monitor additional KPIs to keep track of their institutional performance. These additional KPIs, such as successful application rates, students per class, and revenue streams generated by the programmes, may be monitored through the dashboard shown in Fig. 10.

The prototype dashboard was developed with HF-HEQ-BI FUT. To evaluate the BI systems, both Howson [46] and Dyczkowski et al. [47] adopted a BI scorecard approach. Dashboards capabilities may be evaluated using the BI scorecards by evaluating the design, presentation, alerting, analysis, KPIs, interactivity, delivery, and architecture [41]. Further consultancy work might benefit to determine the requirements in evaluating the BI dashboard and thus improve its performance. This might include incorporating the applications of the BI dashboard in several colleges in the KSA and obtaining their feedback on the use of the dashboard.

The dashboard presented in the paper was developed through the use of HF-HEQ-BI FUT. The dashboard presents the mandatory NCAAA KPIs as well as other KPIs related to the QA factors identified by the HF-HEQ-BI framework. The development of the dashboard went through several iterations of improvement until reaching the presentation style presented in this paper. Fig. 11 outlines the development screen from Microsoft Power BI.

The dashboard design enables additional performance metrics for KPIs to be included and also supports the monitoring of social media streams. Fig. 12 presents the visualisation of the dashboard outputs from FUT.

As illustrated in Fig. 12, the main screen of the dashboard presents the mandatory NCAAA KPIs. The user can navigate to access the supporting documentation for QA activities such as QA manuals or compliance with safety procedures audit logs. The dashboard also allows customisation of any further required visualisation and monitors as many KPIs as the institution’s top management desires. The end user of the dashboard will be allowed to monitor QA performance through the dashboard screen. The users also can drill-down to the data in the dashboard to evaluate the performance of a specific college or university department, as shown in Fig. 13.

Fig. 14 illustrates the filter abilities of the dashboard based on keywords and time. As it can be seen in ‘top-level social media analytics’ in Fig. 14, the sentiment analysis dashboard shows the Twitter analytics for a specified period of time. The user can find the numbers of Tweets, Retweets, and Likes on the dashboard. Then the results can be filtered to show the Tweets related to a specified period of time using the slide bar filter, as shown in ‘period filter applied’ in Fig. 14. As a result, the numbers of Tweets, Retweets, and Likes are changed to reflect the new period specified. In addition, the results also can be filtered to show the Tweets containing specific keywords as shown in ‘keyword search applied’ in Fig. 14. The dashboard reflects the changes showing Tweets, Retweets, and Likes that contain these keywords. The prototype dashboard has been developed to illustrate the way in which the HF-HEQ-BI framework can be used to develop a QA dashboard. Institutions which develop such dashboard based on the framework can customise the presentation of the data to obtain more sophisticated sentiment analyses.

This paper showed how the HF-HEQ-BI framework can be applied to develop a BI prototype dashboard for monitoring quality in HEIs and a dashboard has been developed from the framework. The paper has also outlined different BI architectures that HEIs may adopt to handle the big data challenge while monitoring QA. These architectures show different configurations of BI systems that HEIs may adopt based on their requirements and available resources. The development process of the BI dashboard was discussed to show how HF-HEQ-BI FUT may be used to determine the requirements to be presented in the dashboard. HF-HEQ-BI FUT assists in determining the required visualisation for each KPI to be displayed in the dashboard. FUT is customisable and can be adapted by users to reflect the requirements of individual HEI. FUT shows how decision-makers in HEIs may select various visualisation dashboards while monitoring QA compliance regarding NCCAA standards.

A prototype dashboard was developed to illustrate the application of the framework and also to evaluate the HF-HEQ-BI framework. The developed dashboard shows a monitoring display of the mandatory NCAAA KPIs, institutional specific KPIs, additional KPIs, and social media analytics. The social media analytics presented in the dashboard included the sentiment analysis of data drawn from Twitter through Twitter API. Additionally, the proposed dashboard allows the dashboard users to represent additional KPIs related to performance monitoring in the institutions.

The developed dashboard based on the HF-HEQ-BI framework shows how data retrieved from social media is presented in summarised diagrams to handle the big data challenge. The prototype dashboard is dynamic and KPIs can be added or modified easily based on the changes in regulations or institutional directions of the strategic plans. Our future work will focus on the evaluation of the proposed dashboard to meet the big data challenge.

The authors declare no conflicts of interest.