Saudi Journal of Emergency Medicine

Previous Article Next Article

Original Article

DOI: 10.24911/SJEMed/72-1640256220 View Abstract View PDF
Open access Creative Commons License

Almazroua et al. Saudi J Er Med. 2022;3(2):138-144

Saudi Journal of Emergency Medicine

Psychological and mental wellbeing of healthcare workers in Saudi Arabia during the COVID-19 pandemic

Faisal Y. almazroua1,2 ORCID logo, Abdulaziz Almutairi1,3, Farhan Alanazi1, Ayman Talla1, Mazi Alanazi1*

Correspondence to: Mazi Alanazi

*Department of Emergency Medicine, King Saud Medical City, Riyadh, Saudi Arabia.

Email: moazi-99 [at] hotmail.com

Full list of author information is available at the end of the article.

Received: 23 December 2021 | Accepted: 09 March 2022

© almazroua, Almutairi, Alanazi, Talla, Alanazi.
This is an open access article distributed in accordance with the Creative Commons Attribution (CC BY 4.0) license: (https://creativecommons.org/licenses/by/4.0/) which permits any use, Share — copy and redistribute the material in any medium or format, Adapt — remix, transform, and build upon the material for any purpose, as long as the authors and the original source are properly cited.

ABSTRACT

Background:

The COVID-19 pandemic has induced a substantial burden on healthcare organizations, with increased workload and stress for healthcare providers. Healthcare professionals working on the frontline are vulnerable to stress, putting their psychological and mental wellbeing in considerable jeopardy.

Methods:

This is a cross-sectional study using data collected from a self-administered questionnaire that was distributed electronically to healthcare workers. The validated Kessler Psychological Distress Scale (K10) and the General Self-Efficacy Scale (GSE) were used to measure the level of stress of the healthcare professionals and the psychological impact of the pandemic on them.

Results:

Two hundred and thirteen participants were included. A third of the responders (34.3%) were previously infected with COVID-19, while 72.3% had to self-isolate during the pandemic. The average score for the K10 questionnaire was 24.8 ± 9 out of 50, where almost a third of the responders (33.3%) had very high psychological stress levels. The average score for the GSE was 30.2 ± 5.3 points out of 40. Nurses and physicians (p = 0.033) showed significantly higher scores on the K10 survey. Furthermore, nurses between ages 20 and 29 (p = 0.025) and nurses of any age (p = 0.009) had significantly high levels of psychological stress.

Conclusion:

Healthcare professionals are at significant risk of psychological and mental adverse effects, particularly nurses on the frontline. Such findings should drive initiatives to prepare task forces that can put forward and implement strategies to minimize the risk imposed on the psychological and mental wellbeing of frontline health workers.

Keywords:

COVID-19, Saudi Arabia, frontline, psychological, mental, stress.

Introduction

The novel SARS-CoV-2 virus that causes COVID-19 has resulted in a global pandemic throughout the last year. The viral infection was first detected in China at the end of November 2019 [1]. Transmission of the virus was rapid and progressive not only among the Chinese population but also worldwide [2]. Consequently, the World Health Organization labeled COVID-19 as a pandemic by the first quarter of 2020. The number of admitted patients and the mortality rate have steadily increased globally since this announcement, and the number of deaths due to COVID-19 had approached more than two million by the third quarter of 2020 [3].

Increasing numbers of cases have led to a significant burden upon healthcare organizations, putting healthcare providers under severe pressure, which potentially affects their ability to safely provide health services [4]. Unfortunately, the pandemic has exhausted healthcare in terms of challenges in the supply chain and internal burn-out rate when staff morale has reduced and stress has increased. This may result in patients not being admitted in time or the staff being unable to provide any effective help; moreover, the combination of these alongside other factors such as the struggle with finances may cause a collapse of the healthcare system in some countries [5]. Exhaustion among healthcare workers, particularly those working on the frontline, may quicken the process of this collapse [6].

Multiple factors have contributed to the exhaustion of clinicians and healthcare personnel [7]. Some of the major causes were redeployment to departments with high loads of COVID-19 cases as well as the increased possibility of transmission among healthcare workers; this, in turn, increases the burden on their colleagues as the infected need to be isolated [8]. The COVID-19 crisis has led to an overwhelming mental impact on health care workers, which manifests as severe burnout and anxiety [9].

Recent reports from China and European countries have demonstrated that frontline workers have suffered significant psychological burdens. These are primarily related to their colleagues and family members getting infected in addition to the increased workload [10,11]. In some instances, this psychological impact is severe, leading to significant anxiety or even depression [12].

Despite the information available from western countries on the magnitude of psychological and mental effects on healthcare professionals during the COVID-19 crisis, there is a scarcity of data regarding the mental and psychological effects in Arab countries. Accordingly, this study aimed to assess the magnitude of the mental challenge caused by the pandemic and identify the factors contributing to psychological and mental stress imposed on healthcare professionals during such a time of crisis.

Materials and Methods

Study design

This quantitative study included a self-reported survey distributed to the staff working at the Emergency Department of King Saud Medical City (KSMC) during the COVID-19 pandemic, between March 2020 and August 2020.

Data collection

The survey included questions on both professional and demographic information including age, gender, years of experience, and specialty. Additionally, to evaluate the psychological burden, two validated questionnaires were used. The Kessler Psychological Distress Scale (K10), which is a widely used self-reporting questionnaire, has a 5-point Likert scale ranging from 1 (none of the time) to 5 (all of the time) [13]. It measures psychophysiological, emotional, behavioral, and cognitive manifestations. It is a 10-item measure of the frequency recording the respondent’s history of anxiety, depression, or distress concerning any difficult and challenging situation. Scoring of the K10 survey includes the following: low (10 to 15 points, indicating minimal or no psychological stress), moderate (16 to 21), high (22 to 29), and very high (30 to 50).

To measure the effect on mental wellbeing, a General Self-Efficacy Scale (GSE) was utilized, which identifies emotions and work satisfaction. Scores were given from 1 (not at all true) to 4 (exactly true). Then, the total score is summed for all items. For the GSE, the total score ranges from 10 to 40. It should be recognized that a higher score represents increased self-efficacy [14].

Statistical analysis

All categorical variables were illustrated in the form of frequencies and percentages, while numerical variables were illustrated in the form of means and standard deviations. Comparisons between mean scores with demographic information were made using one-way analysis of variances (ANOVA), where p < 0.05 was considered to be statistically significant. Meanwhile, comparisons for categorical variables were made using chi-square testing at the same level of significance. To minimize missed data, we made sure that all responses by default were completed and that no submissions were allowed before the completion of all elements. Statistical analysis was performed using the Statistical Package for Social Sciences (SPSS) version 23.0 (SPSS Inc., IBM, Armonk, NY).

Results

Two hundred and thirteen workers in KSMC were included in this study. Table 1 describes the demographics of the study population.

Demographic information of workers

Out of the 213 responders, 63.4% were females. Among the workers, 45.5% were between 30 and 39 years old; moreover, 56.8% were married, as illustrated in Table 1.

Professional information of workers

As for the specialty, 57.7% of the responders were nurses; two responders were from the administrative staff. In terms of the education level, 82.6% had a bachelor’s degree, and 62.9% had more than 5 years of work experience (Table 2).

Table 1. Demographic information of workers.

Number Percent
Gender
Female 135 63.4
Male 78 36.6
Age
18 to 29 84 39.4
30 to 39 97 45.5
40 and above 32 15.0
Marital status
Married 121 56.8
Single 92 43.2

Responders were also asked if they were previously infected with COVID-19; almost a third of the responders (34.3%) were previously infected.

Almost three-quarters (72.3%) of the responders had to self-isolate during the COVID-19 pandemic.

Responses to the Kessler Psychological Distress Scale survey

The average score for K10 responses was 24.8 ± 9, with a minimum score of 10 and a maximum score of 50. Almost a third of the responders responded with “some of the time” for all the 10 questions, as illustrated in Table 3.

The K10 questionnaire scores were also classified into low, moderate, high, and very high. Almost a third of the participants (33.3%) were experiencing very high levels of psychological stress, followed by 27.2% experiencing high-stress levels, as demonstrated in Figure 1.

Table 2. Professional and educational information.

Count Percent
Specialty Nurses 123 57.7
Physicians 61 28.6
Emergency medical services 19 8.9
Respiratory therapists 8 3.8
Emergency department administration 2 0.9
Level of education Bachelor’s degree 176 82.6
Diploma 19 8.9
Master’s degree 18 8.5
Work experience Less than 2 years 22 10.3
2 to 5 years 57 26.8
More than 5 years 134 62.9

Responses to the GSE survey

The average GSE score was 30.19 ± 5.3 points, with a minimum score of 11 and a maximum score of 40. The responses demonstrated significant mental stress with almost half of the responses being “moderately true” for all the questions, as shown in Table 4.

Comparison between the different demographic variables and scores for both surveys

The mean scores for both surveys were compared with other demographic variables through one-way ANOVA. Being a nurse or a physician (p = 0.033), having a diploma (p = 0.049), and having 2 to 5 years of work experience (p = 0.031) were all notable factors that showed significantly higher scores on the K10 survey.

In the GSE survey, factors that increased the score included being an emergency medical services worker or a respiratory therapist (p < 0.001) and having had a previous COVID-19 infection (p = 0.041); these were significantly associated with higher GSE scores, as demonstrated in Table 5.

Figure 1. Score classification for the Kessler Psychological Distress Scale survey.

Table 3. Responses to Kessler Psychological Distress Scale questionnaire.

None of the time A little of the time Some of the time Most of the time All of the time
1. During the last 30 days, how often did you feel exhausted for no good reason? 6.6 17.4 38.5 23 14.6
2. During the last 30 days, how often did you feel nervous? 13.6 25.8 33.8 20.2 6.6
3. During the last 30 days, how often did you feel so nervous that nothing could calm you down? 33.8 26.8 29.1 8 2.3
4. During the last 30 days, how often did you feel hopeless? 42.7 23.9 21.6 7.5 4.2
5. During the last 30 days, how often did you feel restless or fidgety? 26.8 23 29.1 14.1 7
6. During the last 30 days, how often did you feel so restless that you could not sit still? 42.7 24.9 20.2 7 5.2
7. During the last 30 days, how often did you feel depressed? 27.2 26.8 31 9.9 5.2
8. During the last 30 days, how often did you feel that everything was an effort? 12.2 21.1 38.5 18.3 9.9
9. During the last 30 days, how often did you feel so sad that nothing could cheer you up? 25.4 29.6 24.9 12.7 7.5
10. During the last 30 days, how often did you feel worthless? 36.6 24.4 25.8 8.9 4.2

Table 4. Responses to the GSE survey.

Not at all true Hardly true Moderately true Exactly true
1. I can always manage to solve difficult problems if I try hard enough. 4.2 17.8 48.8 29.1
2. If someone opposes me, I can find the means and ways to get what I want. 8.9 21.6 51.2 18.3
3. It is easy for me to stick to my aims and accomplish my goals. 3.3 17.8 53.5 25.4
4. I am confident that I can deal efficiently with unexpected events. 3.8 14.1 55.4 26.8
5. Thanks to my resourcefulness, I know how to handle unforeseen situations. 0.5 20.2 55.4 23.9
6. I can solve most problems if I invest the necessary effort. 2.3 19.2 43.2 35.2
7. I can remain calm when facing difficulties because I can rely on my coping abilities. 2.3 19.2 49.3 29.1
8. When I am confronted with a problem, I can usually find several solutions. 3.3 17.8 52.6 26.3
9. If I am in trouble, I can usually think of a solution. 2.3 16.9 49.8 31
10. I can usually handle whatever comes my way. 3.8 20.7 46.5 29.1

Table 5. Factors influencing mental and psychological stress during the COVID-19 pandemic.

K10 p-value GSE p-value
Mean SD Mean SD
Gender Male 24.9 9.0 0.982 29.9 5.6 0.597
Female 24.8 9.0 30.3 5.2
Age 18 to 29 26.6 9.5 0.071 29.8 5.3 0.569
30 to 39 23.8 8.5 30.2 5.6
40 and above 23.5 8.8 31.0 5.0
Marital status Single 24.5 8.8 0.600 31.0 4.8 0.062
Married 25.1 9.2 29.6 5.7
Specialty Nurses 25.5 9.3 0.033* 30.0 5.7 < 0.001*
Physicians 25.7 8.7 28.6 4.2
Paramedics 19.8 8.0 34.4 3.9
Respiratory therapists 20.5 6.4 34.8 3.7
Emergency department administrators 24.5 6.4 31.5 2.1
Education Bachelor’s 24.5 9.0 0.049* 30.5 5.4 0.231
Diploma 27.0 8.8 29.5 6.0
Master’s 25.6 9.5 28.3 3.6
Years of work experience Less than 2 years 25.6 8.6 0.031* 29.3 3.3 0.578
2 to 5 years 26.9 9.1 29.9 6.0
More than 5 years 23.8 8.9 30.5 5.4
Previous COVID-19 infection Yes 24.2 8.5 0.460 31.2 4.6 0.041*
No 25.2 9.3 29.7 5.6
Self-isolation Yes 25.0 9.0 0.728 30.4 5.2 0.342
No 24.5 9.3 29.6 5.8

* means significant p value < 0.05

The K10 scores (low, moderate, high, and very high psychological stress) were compared to different demographic and professional variables using Chi-square testing at a significance level of p < 0.05.

We showed that being female (p = 0.022), 18 to 29 years old (p = 0.025), a nurse (p = 0.009), and having higher education levels (p = 0.041) were all factors that significantly contributed to very high levels of psychological stress during the COVID-19 pandemic, as shown in Table 6.

Discussion

The COVID-19 pandemic has imposed a significant workload on hospital nursing staff, medical specialists, and administrative workers [15]. The increased number of admitted patients increases the risk of hospital staff becoming infected or required to self-isolate; moreover, it also raises the risk of potentially transmitting the infection to their families [16]. All these factors have contributed to significant stress for these workers, which is underexamined in the Gulf area [17].

This investigation measured the psychological and mental stress levels of Saudi healthcare workers in a hospital in Saudi Arabia during the COVID-19 pandemic. The study demonstrated that both medical and administrative staff significantly suffered from very high risks of psychological and mental distress.

Table 6. Factors influencing a very high level of psychological stress.

Low Moderate High Very high p-value
Gender Male 17.9% 19.2% 34.6% 28.2% 0.022*
Female 15.6% 25.2% 23.0% 36.3%
Age 18 to 29 10.7% 23.8% 28.6% 36.9% 0.025*
30 to 39 17.5% 25.8% 27.8% 28.9%
40 and above 28.1% 12.5% 21.9% 37.5%
Marital status Single 15.2% 26.1% 22.8% 35.9% 0.516
Married 17.4% 20.7% 30.6% 31.4%
Specialty Nurses 13.0% 26.0% 22.0% 39.0% 0.009*
Physicians 16.4% 14.8% 36.1% 32.8%
Paramedics 36.8% 26.3% 26.3% 10.5%
Respiratory therapists 25.0% 25.0% 37.5% 12.5%
Emergency department administration 0.0% 50.0% 50.0% 0.0%
Education Bachelor’s 16.5% 26.1% 25.0% 32.4% 0.041*
Diploma 5.3% 15.8% 47.4% 31.6%
Master’s 27.8% 0.0% 27.8% 44.4%
Years of work experience Less than 2 years 13.6% 22.7% 27.3% 36.4% 0.667
2 to 5 years 10.5% 19.3% 33.3% 36.8%
More than 5 years 13.6% 22.7% 27.3% 36.4%
Previous COVID-19 infection Yes 21.9% 16.4% 30.1% 31.5% 0.206
No 13.6% 26.4% 25.7% 34.3%
Self-isolation Yes 15.6% 22.7% 28.6% 33.1% 0.888
No 18.6% 23.7% 23.7% 33.9%

Psychological stress and mental wellbeing have previously been examined among healthcare professionals in different contexts. Li et al. [18] have assessed the psychological distress that frontline nurses have suffered during the COVID-19 pandemic; they have demonstrated that frontline nurses have significantly higher stress levels in comparison to those of the nurses working on non-frontline services (p < 0.001). This lower stress level was contributed to a psychological training program given to frontline nurses [18].

In this investigation, more than half of the individuals in the included cohort were females and nurses. It has been revealed that females and nurses are at a significantly higher risk of having very high levels of psychological stress (p = 0.002 and 0.033, respectively). Accordingly, a psychological training program proposed by Li et al. [18] recommends a viable solution for the medical staff, particularly nurses, to reduce their psychological stress.

Furthermore, Kang et al. [19] have illustrated that the most common type of psychological stress among physicians is anxiety in a mild form, representing 34.4% of the responders. However, Kang et al. [19] have also shown that almost half of the healthcare workers have access to some psychological support materials through media, which help to decrease their psychological burden [19].

Huang et al. [20] have demonstrated a higher anxiety level among nurses than among the doctors (p = 0.039). Furthermore, Huang et al. [20] have also demonstrated that females are at a significantly high risk of increased anxiety (p < 0.045), while 16.09% of the healthcare professionals have mild psychological distress [20].

Chung et al. [21] conducted a survey in which psychological and emotional stress experienced by frontline physicians and hospital staff was examined. Chung et al. [21] have revealed that frontline physicians have higher mental and psychological levels of stress as they fear they will become infected and transmit the disease to their families. These findings are similar to the results of our study, which demonstrate that nurses and physicians are at significant risk of the highest levels of stress.

It is important to understand that this study suffered from some limitations. Due to the self-reported questionnaire technique used in this study, the participants’ responses were solely dependent on their subjective opinions, which could affect the outcome reliability. Furthermore, this survey investigated a single center in Saudi Arabia, which makes external validity questionable and difficult. Furthermore, the professionals who attended psychological training or attempted to reduce their COVID-19 crisis-induced stress were too few for us to be able to compare and assess the usefulness of such solutions.

Conclusion

Healthcare workers and the administrative staff working on the frontline in Saudi hospitals during the COVID-19 pandemic have suffered from significantly high levels of psychological and mental stress. These findings should drive initiatives and regulations to mitigate the impact of such disasters, decrease stress levels, and keep the morale as high and positive as possible. Future studies are also required to measure the correlations between the psychological and mental wellbeing of the working staff and the impact on the provided healthcare services. Moreover, more studies are needed to evaluate the mental and psychological well-being of healthcare workers in other areas of Saudi Arabia.

List of Abbreviations

GSE General Self-Efficacy Scale
ANOVA One-way analysis of variances
SPSS Statistical Package for Social Sciences
KSMC King Saud Medical City

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Funding

None.

Consent to participate

Written informed consent was obtained from all the participants.

Ethical approval

Ethical approval was granted by Institutional Review Board via reference no. H1R1-16-Jul20-01.

Author details

Faisal Y. almazroua1,2, Abdulaziz Almutairi1,3, Farhan Alanazi1, Ayman Talla1, Mazi Alanazi1

  1. Department of Emergency Medicine, King Saud Medical City, Riyadh, Saudi Arabia
  2. Executive Administration of Medical Affairs, King Saud Medical City, Riyadh, Saudi Arabia
  3. Internal Audit Department, King Saud Medical City, Riyadh, Saudi Arabia

References

  1. Wilson W, Raj JP, Rao S, Ghiya M, Nedungalaparambil NM, Mundra H, et al. Prevalence and predictors of stress, anxiety, and depression among healthcare workers managing COVID-19 pandemic in India: a nationwide observational study. Indian J Psychol Med. 2020;42(4):353–8. https://doi.org/10.1177/0253717620933992
  2. Lu W, Wang H, Lin Y, Li L. Psychological status of medical workforce during the COVID-19 pandemic: a cross-sectional study. Psych Res. 20204;288:112936. https://doi.org/10.1016/j.psychres.2020.112936
  3. Shaukat N, Ali DM, Razzak J. Physical and mental health impacts of COVID-19 on healthcare workers: a scoping review. Int J Emerg Med. 2020;13(1):40. https://doi.org/10.1186/s12245-020-00299-5
  4. Cao J, Wei J, Zhu H, Duan Y, Geng W, Hong X, et al. A study of basic needs and psychological wellbeing of medical workers in the fever clinic of a tertiary general hospital in Beijing during the COVID-19 outbreak. Psychother Psychosomat. 2020;89(4):252–4. https://doi.org/10.1159/000507453
  5. Galbraith N, Boyda D, McFeeters D, Hassan T. The mental health of doctors during the COVID-19 pandemic. BJPsych Bull. 2020;28:1–4. https://doi.org/10.1192/bjb.2020.44
  6. Huang J, Liu F, Teng Z, Chen J, Zhao J, Wang X, et al. Care for the psychological status of frontline medical staff fighting against Coronavirus Disease 2019. Clin Infect Dis. 2020;71(12):3268–9. https://doi.org/10.1093/cid/ciaa385
  7. Greenberg N, Docherty M, Gnanapragasam S, Wessely S. Managing mental health challenges faced by healthcare workers during COVID-19 pandemic. BMJ. 2020;368:m1211. https://doi.org/10.1136/bmj.m1211
  8. Ho CS, Chee CY, Ho RC. Mental health strategies to combat the psychological impact of COVID-19 beyond paranoia and panic. Ann Acad Med Singap. 2020;49(3):155–60. https://doi.org/10.47102/annals-acadmedsg.202043
  9. Azoulay E, De Waele J, Ferrer R, Staudinger T, Borkowska M, Povoa P, et al. Symptoms of burnout in intensive care unit specialists facing the COVID-19 outbreak. Ann Intens Care. 2020;10(1):110. https://doi.org/10.1186/s13613-020-00722-3
  10. Vagni M, Maiorano T, Giostra V, Pajardi D. Hardiness, stress and secondary trauma in Italian healthcare and emergency workers during the COVID-19 pandemic. Sustainability. 2020;12(14):5592. https://doi.org/10.3390/su12145592
  11. Gupta S, Kohli K, Padmakumari P, Dixit PK, Prasad AS, Chakravarthy BS, et al. Psychological health among armed forces doctors during COVID-19 pandemic in India. Ind J Psychol Med. 2020;42(4):374–8. https://doi.org/10.1177/0253717620934037
  12. Cai H, Tu B, Ma J, Chen L, Fu L, Jiang Y, et al. Psychological impact and coping strategies of frontline medical staff in Hunan between January and March 2020 during the outbreak of coronavirus disease 2019 (COVID 19) in Hubei, China. Med Sci Monit. 2020;26:e924171. https://doi.org/10.12659/MSM.924171
  13. Kessler Psychological Distress Scale. Available from: https://www.statisticssolutions.com/kessler-psychological-distress-scale-k10/
  14. Schwarzer R, Jerusalem M. Generalized Self-Efficacy scale. In: Weinman J, Wright S, Johnston M, editors. Measures in health psychology: a user’s portfolio. Causal and control beliefs. Windsor, UK: NFER-NELSON; 1995. pp 35–7. https://doi.org/10.1037/t00393-000
  15. Walton M, Murray E, Christian MD. Mental health care for medical staff and affiliated healthcare workers during the COVID-19 pandemic. Eur Heart J. 2020;9(3):241–7. https://doi.org/10.1177/2048872620922795
  16. Al Aseri ZA, Suriya MO, Hassan HA, Hasan M, Sheikh SA, Al Tamimi A, et al. Reliability and validity of the Hospital Anxiety and Depression Scale in an emergency department in Saudi Arabia: a cross-sectional observational study. BMC Emerg Med. 2015;15:28. https://doi.org/10.1186/s12873-015-0051-4
  17. Braquehais MD, Vargas-Cáceres S, Gómez-Durán E, Nieva G, Valero S, Casas M, et al. The impact of the COVID-19 pandemic on the mental health of healthcare professionals. QJM. 2020;113(9):613–7. https://doi.org/10.1093/qjmed/hcaa207
  18. Li Z, Ge J, Yang M, Feng J, Qiao M, Jiang R, et al. Vicarious traumatization in the general public, members, and non-members of medical teams aiding in COVID-19 control. Brain Behav Immun. 2020;88:916–9. https://doi.org/10.1016/j.bbi.2020.03.007
  19. Kang L, Ma S, Chen M, Yang J, Wang Y, Li R, et al. Impact on mental health and perceptions of psychological care among medical and nursing staff in Wuhan during the 2019 novel coronavirus disease outbreak: a cross-sectional study. Brain Behav Immun. 2020;87:11–7. https://doi.org/10.1016/j.bbi.2020.03.028
  20. Huang JZ, Han MF, Luo TD, Ren AK, Zhou XP. Mental health survey of 230 medical staff in a tertiary infectious disease hospital for COVID-19. Zhonghua Lao Dong Wei Sheng Zhi Ye Bing Za Zhi. 2020;38(3):192–5.
  21. Chung JPY, Yeung WS. Staff mental health self-assessment during the COVID-19 outbreak. East Asian Arch Psych. 2020;30(1):34. https://doi.org/10.12809/eaap2014