ARTÍCULOS

 

Psychometric properties of Colquitt’s Organizational Justice Scale in Argentine workers

Propiedades psicométricas de la Escala de Justicia Organizacional de Colquitt en trabajadores argentinos

 

Alicia Omar^a,*, Solana Salessi^b , Juan Diego Vaamonde^a, Florencia Urteaga^b

^aInstituto de Investigaciones de la Facultad de Humanidades y Artes, Universidad Nacional de Rosario; Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Argentina

^bFacultad de Ciencias de la Gestión, Universidad Autónoma de Entre Ríos, Argentina

*agraomar@yahoo.com

---

Abstract

Organizational justice refers to workers’ perceptions of what is fair and unfair in their workplaces. The aim of the present study was to provide evidence of the factorial validity and psychometric properties of the Argentine version of Colquitt’s Organizational Justice Scale. To achieve this objective, an empirical, quantitative, instrumental study with a cross-sectional design was conducted. Data were obtained from a convenience sample of 406 employees (212 men and 194 women) of different Argentine organizations. Confirmatory factor analysis results supported the four-factor structure of the scale (distributive, procedural, interpersonal, and informational justice). The instrument showed good reliability indices (ordinal alpha and composite reliability values greater than .80) and adequate discriminant and convergent validity (average variance extracted indices greater than .60). These psychometric characteristics make the validated scale a useful tool to measure justice perceptions within Argentine organizations.

Keywords: justice, organizations, confirmatory factor analysis, validity, Argentine workers

---

Resumen

La justicia organizacional se refiere a las percepciones de los trabajadores sobre lo que es justo e injusto en su trabajo. El objetivo del presente estudio fue presentar evidencias sobre la validez factorial y propiedades psicométricas de la versión argentina de la Escala de Justicia Organizacional de Colquitt. Se condujo una investigación empírica, cuantitativa, instrumental, de corte transversal. Se contó con una muestra por disponibilidad de 406 trabajadores (212 varones y 194 mujeres) de organizaciones argentinas. Los resultados del análisis factorial confirmatorio corroboraron la estructura tetrafactorial de la escala (justicia distributiva, procedimental, interpersonal e informacional). Se obtuvieron adecuados índices de confiabilidad (valores alfa ordinal y de confiabilidad compuesta mayores de .80) así como de validez discriminante y convergente (índices de varianza media extraída superiores a .60). Tales características psicométricas transforman a la escala validada en una herramienta útil para medir las percepciones de justicia al interior de las organizaciones argentinas.

Palabras clave: justicia, organizaciones, análisis factorial confirmatorio, validez, trabajadores argentinos

---

Introducción

The term organizational justice (OJ) was coined by Greenberg (1987) to refer to employees’ perceptions about what is fair and what is unfair in their workplaces. The concept entails a personal assessment of the ethical and moral standards that characterize the organization. In their analysis of the origins of OJ, Cropanzano, Bowen, and Gilliland (2007) distinguish between the prescriptive approach of philosophers and the descriptive approach of social scientists. Philosophers have discussed the issue of justice long before social scientists, trying to determine what kinds of actions are truly fair. This is the prescriptive approach, which can still be found in the domain of business ethics. In contrast, the interest of social scientists has been in what people think is fair. This constitutes the descriptive approach, which tries to understand why people perceive certain events as fair and others as unfair. From this perspective, justice is a subjective and descriptive concept that captures what the individual believes is fair, rather than an objective reality or prescriptive moral code. In this paper, OJ will be addressed from the viewpoint of social scientists.

The study of OJ has gained increasing interest for its impact on both workers’ well-being and organizational functioning. A substantial amount of research supports the claim that while perceived justice contributes to building commitment and confidence (Ohana, 2014), increases organizational performance and citizenship behaviors (Schilpzand, Martins, Kirkman, Lowe, & Chen, 2013), and promotes satisfaction and welfare (Cassar & Buttigieg, 2015), perceived injustice provokes undesirable consequences, such as negative attitudes and emotions, turnover intentions, and counterproductive work behaviors (Colquitt et al., 2013; Proost, Verboon, & van Ruysseveldt, 2015).

In particular, organizational injustice acts as an occupational stressor that may lead to serious individual and organizational problems (Colquitt et al., 2013; Nasurdin, Ahmad, & Razalli, 2014). This association between organizational injustice and stress has been explained in terms of the effort-reward imbalance model (Siegrist, 1996), since unfair distribution of rewards, unfair decision processes, and unfair interpersonal treatment violate core expectations about reciprocity and exchange at work (Nasurdin et al., 2014). In fact, Elovainio, Heponiemi, Sinervo, and Magnavita (2010) have shown that the lack of justice is linked to greater risks of suffering psychological strain, psychiatric disorders, illness-related work absences, and sleep disorders. Also, low OJ may contribute to serious health problems, such as cardiovascular disease, gastritis, and ulcers, all conditions intimately related to occupational stress. In view of this wide repertoire of consequences of (in)justice in organizations, it is important to have a valid and reliable instrument, adapted to the Argentine organizational context, in order to explore workers’ justice perceptions and act accordingly.

OJ as a multidimensional construct

Since the appearance of the foundational studies on OJ, researchers have debated about its structure and dimensions. In light of recent meta-analytic findings (Colquitt et al., 2013), there is now consensus among experts about the multidimensionality of the OJ construct, which comprises the dimensions of distributive justice, procedural justice, interpersonal justice, and informational justice.

Distributive justice refers to perceptions of fairness in relation to the distribution of outcomes (salary, promotions and rewards), and it specifically addresses the degree to which outcomes are equitable (Colquitt et al., 2013). Given its emphasis on benefits, distributive justice is mainly related to cognitive, affective, and behavioral reactions directed to particular outcomes (Ambrose & Schminke, 2009).

Procedural justice indicates perceptions of fairness in relation to the means, mechanisms, and processes by which the benefits and rewards are distributed in the organization (Leventhal, 1980). While distributive justice perceptions are related to satisfaction with individual results, procedural justice perceptions are linked to attitudes and behaviors that are relevant to the organization. Hence low procedural justice elicits intellectual and emotional indignation, which translates into resentment and lack of cooperation (Cho & Sai, 2013).

Interactional justice (Bies & Moag, 1986) corresponds to the human side of OJ and it was defined as the way directors and managers treat employees during the application of procedures. Greenberg (1993), argued that the social components of interactional justice had to be broken down into two separate types of justice: interpersonal justice and informational justice. Interpersonal justice reflects the degree to which people are treated with politeness, dignity, and respect by authorities or third parties involved in executing procedures or determining outcomes. Informational justice focuses on the truthfulness and adequacy of the information and explanations provided to employees regarding the distribution of outcomes and/or the implementation of procedures. In the case of interpersonal justice, fair treatment is expected to increase the degree of acceptance of decisions and to promote other types of positive reactions among personnel. In the case of informational justice, the explanations and justifications provided to employees are presumed to mitigate their negative reactions to injustice perceptions and/or to the inequities in the allocation of results (Colquitt et al., 2013; Greenberg, 1993).

The measurement of OJ

As considerable debate on the conceptual definition of OJ was taking place, efforts to find suitable measures to explore the construct were increasing. Different instruments were designed to examine either general perceptions of justice (Ambrose & Schminke, 2009) or any of its dimensions (Sweeney & McFarlin, 1997). Despite the profusion of available techniques, the literature review shows that the OJ Scale developed by Colquitt (2001), which measures the four dimensions proposed by Greenberg (1993), is one of the most widespread justice measures. Colquitt integrated his scale with 20 five-point Likert-type items, which were distributed as follows: 7 items that measure procedural justice (taken from Thibaut & Walker, 1975, and from Leventhal, 1980), 4 items that measure distributive justice (taken from Leventhal, 1976), 4 items that measure interpersonal justice (taken from Bies & Moag, 1986), and 5 items that measure informational justice (taken from Bies & Moag, 1986 and from Shapiro, Buttner, & Barry, 1994). This scale was validated by Colquitt in two different studies: one carried out with 301 students from a management course, and another one executed with 337 employees from two plants in a leading automobile parts manufacturing company. In the study conducted with students, OJ was examined in the context of university education, so distributive justice concerned the fairness of the grades students had received, procedural justice involved the fairness of the decision-making processes used by the instructors, and interactional justice was related to the instructors’ interpersonal treatment of students. In the study conducted with employees, distributive justice concerned the fairness of the outcomes employees received from their work (e.g., pay, raises, rewards and promotions), procedural justice involved the fairness of the decision-making procedures used by the supervisors, and interactional justice was related to the supervisors’ interpersonal treatment received by employees. Both studies demonstrated the four-factor structure of the instrument, and thus empirically corroborated the theorized multidimensionality of the OJ construct. These studies also provided evidence about the reliability of the subscales (alpha coefficients ranged from .78 to .93), as well as the predictive validity and the nomological network of the construct.

In recent years, multiple validations of this scale have been performed around the world, among which the German, Italian, Japanese, Norwegian, Spanish, and Puerto Rican validations stand out. In this regard, the German validation (Maier, Streicher, Jonas, & Woschée, 2007), which was carried out on a sample of 227 employees of different occupations, reproduced the original four-factor structure communicated by Colquitt. The authors obtained good fit indices, interfactor correlations in the range between .30 and .60, and Cronbach’s alpha coefficients ranging between .79 and .93. The Italian validation (Di Fabio, 2008), which was conducted on a sample of 405 hospital employees, also reproduced the factor solution initially obtained by Colquitt. In this case, reliability coefficients were higher than .80 for the four identified factors, and a very good concurrent validity was obtained. Regarding the Japanese validation (Shibaoka et al., 2010), which was based on data from a multi-occupational sample of 229 employees, it also yielded an adequate four-factor structural solution. The authors observed significant correlation coefficients between the four justice dimensions and the scores of effort-reward imbalance, psychological distress, and job satisfaction, all of which demonstrate good construct validity. Besides, Cronbach’s alpha coefficients were above .90 and intraclass correlation coefficient was .91, which provided evidence of high internal consistency and temporal stability of the scores of this Japanese version. As to the Norwegian scale validation (Olsen, Myrseth, Eidhamar, & Hystad, 2012), which was carried out with 312 army officers, it achieved adequate internal consistency and a good fit between the sample data and a four-factor model integrated by the dimensions of distributive, procedural, interpersonal, and informational justice. The correlations between the subscales varied between .44 and .72, and the correlations between the justice subscales and self-sacrifice behaviors reached significant values between .15 and .25, demonstrating construct validity.

The Spanish validation (Díaz-Gracia, Barbaranelli, & Moreno-Jiménez, 2014), which was subsequent to some Latin American novel research (Mladinic, 2002; Omar, Oggero, Maltaneres, & Paris, 2003), was based on data from 460 employees of the service sector. In line with most of the validations reviewed, Díaz-Gracia et al. (2014) found a four-factor solution similar to Colquitt’s, and reported interfactor correlations ranging from .40 to .71, composite reliability indices between .88 and .94, and moderate correlations between OJ, job satisfaction, and workplace incivility. Finally, the Puerto Rican validation (Rodríguez-Montalbán, Martínez-Lugo, & Sánchez-Cardona, 2015), which was conducted on a sample of 383 workers, mostly employees of the private sector, also showed an adequate fit of the data to a four-factor structure. The four subscales presented good reliability indices (Cronbach’s alphas between .88 and .94, and composite reliability between .89 and .94). Convergent validity, which was explored through the average variance extracted (AVE), was in the range of .51 - .60, and the correlations of the subscales with work engagement reached values between .44 and .48, which offers evidence of the concurrent validity of the validated version. Considering these antecedents, and with the purpose of having a suitable instrument to measure OJ perceptions among Argentine workers, the objective of this study was to analyze the factor structure and to determine the validity and reliability of the Argentine version of Colquitt’s OJ Scale.

Method

Study design

The present study can be regarded as an empirical, quantitative, descriptive, cross-sectional research. It falls into the category of instrumental studies (Ato, López, & Benavente, 2013), as it examines the psychometric properties of an OJ measure. To accomplish the research objective, a pilot study and a main study were carried out.

Pilot study

The aim of this pilot study was to translate Colquitt’s OJ Scale from English into Spanish and to adapt it for use with Argentine samples. In this instance, the analyses of the semantic and operational equivalences of the instrument were performed in accordance with the guidelines provided by Muñiz, Elosua, and Hambleton (2013). The semantic equivalence consists in the translation of the items, keeping the meaning between different languages. In this case, the translation and back-translation of the items were done by two English translators. The operational equivalence refers to the maintenance of operational characteristics before and during the application of the instrument, especially in relation to the time it takes to complete, the clarity of instructions to perform the task, and the syntactic and semantic appropriateness of the items. To achieve the aforementioned objective, the Spanish prototypical version of Colquitt’s scale, with a 5-point Likert format varying from 1 (never) to 5 (always), was applied to a convenience sample of 120 Argentine workers (53% men; M_age = 33.81, SD_age = 5.64; M_tenure = 4.74, SD_tenure = 2.24). All participants completed the scale in their respective workplaces, during specific hours assigned by the organizations for training and/ or research purposes. In all cases, once the administration of the scale had been completed, some minutes were allotted for employees to express their opinions about the clarity of the items, possible ambiguities, the amount of time required to answer them, and similar semantic and operational aspects.

Main study

The aim of this study was to analyze the psychometric properties of the translated and adapted version of the OJ Scale (Colquitt, 2001).

Participants

Following the recommendation that instrumental studies should have at least 200 cases to ensure a stable and generalizable factor solution (Lloret, Ferreres, Hernández, & Tomás, 2017), an initial nonrandom sample of 428 employees was taken. Twentytwo cases had to be discarded for not having fully completed the research questionnaire, so the final sample comprised 406 workers (212 men and 194 women) of various organizations located in the central region of Argentina (41% from the province of Santa Fe, 30% from Entre Ríos, and 29% from Buenos Aires). The mean age of the participants was 36 years (SD = 9.25), and the mean tenure was 7 years (SD = 5.78). Fifty-two percent had secondary education level, and 48% had completed higher education (tertiary and/or university level). Of those sampled, 36% worked in industry, 30% in trade/business, 22% in service companies (banks, insurance companies, consulting firms), and the remaining 12% belonged to educational institutions. Fifty-three percent belonged to the private sector and 47% to the public sector. Regarding the position held, 58% and 42% were hired and permanent employees, respectively, while 11% were managers or supervisors.

Instrument

OJ was examined through the adapted Argentine version of Colquitt’s Scale (2001). This measure consists of 20 items (Table 1) with a Likert-type response format ranging from 1 (never) to 5 (always). Table 1 presents the full adapted version of the instrument.

Procedure

Data collection took place in the second half of 2016. Invitations to participate in the study were sent to the human resource managers of the organizations that accepted to collaborate with the research. Managers were responsible for sending the invitations to employees who met the following inclusion criteria: (1) having tenure of at least two years; (2) being under the direction or supervision of a superior; and (3) receiving a salary or pay for their work. Of a total of 870 invitations sent, 428 employees finally accepted to participate in the study (49.2% response rate). Participation was anonymous and voluntary, and no incentives were offered to employees. Data collection was conducted during working hours at the physical spaces provided by the organizations for that purpose. Participants completed, either individually or in small groups, a booklet containing the study objectives, a set of detailed instructions, and the OJ Scale. The implementation of the research was conducted in full accordance with the ethical guidelines recommended not only by the American Psychological Association’s Ethical Principles of Psychologists and Code of Conduct (2017), but also by the Consejo Nacional de Investigaciones Científicas y Técnicas for Social Sciences and Humanities (CONICET, 2006).

Data analysis

Prior to statistical analysis, the data set was screened for missing values. Then, means, standard deviations, and skewness and kurtosis indices were calculated for all items. Discrimination indices were also computed by means of corrected item-total correlations. In order to evaluate sampling adequacy, Kaiser-Meyer-Olkin measure and Bartlett’s test of sphericity were calculated. Mardia’s normalized multivariate kurtosis estimate was computed to determine multivariate normality (Bentler, 2006).

To establish the structural validity of the scale, a confirmatory factor analysis (CFA) was performed on the polychoric matrix. Following the rival models strategy, Colquitt’s four-factor model was compared with two alternative models. On the one hand, a univariate model in which all items were loaded onto a single latent factor. On the other hand, a three-factor model in which the items of interpersonal justice and informational justice were loaded onto one factor, whereas the items of distributive justice and procedural justice were loaded onto two different factors.

The relevance of applying the rival models strategy for structural analysis was based on both theoretical and empirical-methodological arguments. In this sense, a one-factor model was tested considering the explanatory framework provided by the interaction perspective, which proposes an analysis of the joint effects of different types of justice. Greenberg (1987) argued that the components of OJ interact and «work» together, and pointed out that if organizations manage to maintain at least one of the components of OJ, they can lessen the effects of injustice and even get benefits. The three-factor model was estimated considering interactional justice as the third type of justice, which subsumes both informational and interpersonal justice (Bies & Moag, 1986). The inclusion of this model was based on the fact that many researchers (Bies & Shapiro, 1987; Sweeney & McFarlin, 1997) have not only used different measures of distributive, procedural, and interactional justice, but have also pointed out that each type of justice affects different outcomes and factors. In addition to these theoretical and empirical considerations, Colquitt and his collaborators (Colquitt, 2001; Colquitt, Greenberg, & Zapata-Phelan, 2005; Colquitt & Shaw, 2005; Colquitt, Wesson, Porter, Conlon, & Ng, 2001) have exhorted to contrast structures of 1, 2, 3, and 4 factors, given that such solutions reflect the different ways in which OJ has been conceptualized and measured.

The maximum likelihood estimation method was used together with the Satorra-Bentler (S-B) scaled chi-square, which is recommended when the data come from ordinal scales (Bentler, 2006). With the aim of evaluating the goodness-of-fit of each model, the following conditions were examined: that the SBχ² correction divided by the degrees of freedom (SBχ²/ df) was less than 3, that the Parsimonious Goodness of Fit Index (PGFI) was between .50 and .70, that the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI) were equal to or higher than .90, and that the Root Mean Square Error of Approximation (RMSEA) was less than .05. Also, the AIC (Akaike Information Criterion) index was calculated, knowing that the lower the value, the more parsimonious the model is (Kline, 2013). While convergent loadings –corresponding to the saturations of the items in their respective latent factors– were freely estimated, divergent loadings –corresponding to the saturations in the remaining factors– were set at .30, as moderate factorial complexity was expected. The strength of the divergent loadings was examined by means of structural coefficients, that is, by the correlations of the observable variables (items) and the latent variables (Graham, Guthrie, & Thomson, 2003).

Reliability and validity analyses were performed on the measurement model suggested by the CFA. Given the ordinal nature of the scale, reliability was determined by calculating the composite reliability coefficient. Values greater than .70 indicate satisfactory reliability (Gadermann, Guhn, & Zumbo, 2012). Convergent validity, i.e. the common variance between the observable indicators and the latent variable, was verified by the Average Variance Extracted (AVE). Values greater than .50 are considered evidence of adequate convergent validity, since they indicate that more than 50% of the factor variance is due to their indicators (Bagozzi & Yi, 2012). To assess discriminant validity, specialists (Henseler, Ringle, & Sarstedt, 2015) recommend calculating the square root of the AVE. If the value obtained for each latent variable is greater than the correlation between this variable and the other variables included in the model, it can be assumed that each factor shares more variance with its indicators than with the others.

Given the heterogeneity of the sample, the dimensions of OJ were analyzed according to participants’ gender and occupation. For this reason, analyses of variance (one-way ANOVA) and mean difference tests (Student’s t-tests) were performed, as appropriate. Also, the configural invariance and the strict invariance between men and women, as well as between the different organizational activities (industry, trade/business, services, and education), were examined. For this purpose, an unrestricted base model was initially estimated, followed by alternative models with parameter restrictions (main and crossfactor loadings, intercepts, variances and covariances). The different nested models were compared in relation to variations in their goodness-of-fit indices. Increases in CFI and TLI equal to or less than .010 and increases in RMSEA equal to or less than .015 were deemed to be evidence of invariance (Cheung & Rensvold, 2002). Data processing and analysis were performed using SPSS 22.0, Factor 10.8, and EQS 6.1.

Results

Pilot study

Semantic equivalence. The semantic equivalence analysis was carried out in three steps: the original instrument was translated from English to Argentine Spanish; two English translators back-translated the Argentine version to English; the same translators blindly compared the two versions of the instrument in order to identify the degree of agreement between the original items and the translated ones, according to four levels of equivalence: unaltered, slightly altered, quite altered, and completely altered (Table 2). Considering the general meaning of the items, both professionals indicated that the semantic correspondence between the original and the translated items was highly satisfactory. Although there were small differences in the literal translation of three items (3, 4, and 9), the specialists agreed that the general meaning remained the same between the two versions of the scale. They therefore concluded that the Argentine version of the OJ Scale had adequate semantic equivalence.

Operational equivalence. Before the pilot study, issues such as administration format, time needed to complete the scale, and clarity of the instructions and items were revised by three experts in psychometrics. All of these aspects remained the same with respect to the original scale developed by Colquitt, except for the question included in each section («To what extent ...?») and the associated 5-point Likert response scale (1 = to asmall extent to 5 = to a large extent), which were replaced by «How often...?» («¿Con qué frecuencia...?») and 1 = never (nunca) to 5 = always (siempre), respectively. These modifications were introduced to avoid confusion and to enhance instruction understanding in Spanish language. Also, to clarify the term «outcomes» in the distributive justice section, the phrase «your salary, promotions, rewards, etc.» was added in parentheses. The same was done to elucidate the original phrase «the authority figure who enact the procedures» by specifying «your boss, supervisor, the person responsible for the department, etc.» in parentheses.

Participants in the pilot study indicated that the instructions were intelligible, that they had no problems in understanding the content of the items, and that the 5-point Likert scale did not present any difficulty for them. Even both the group and the individual administration yielded satisfactory results for the adapted version of the scale.

Main study

Preliminary analysis of the items. Given that preliminary data analysis revealed the existence of less than 2% of missing values, median imputation was used for their replacement. Table 3 shows the descriptive statistics, the distribution of the data, and the item discrimination indices.

As can be seen in Table 3, 16 of the 20 items showed skewness and kurtosis values within the recommended range of +1.00 and -1.00. Only four items (12, 13, 14, and 16) had skewness and/or kurtosis values slightly higher. All items presented positive correlations. The KMO sampling adequacy measure was .86, and the Bartlett’s sphericity test was statistically significant (χ² _{(406, 190)} = 5172.56, p < .001). These results indicated that a CFA could be performed on the collected data. However, the Mardia’s normalized multivariate kurtosis estimate was 4.12, exceeding the recommended range of -3 to 3 to assume multinormality (Bentler, 2006). Such observation warranted the use of robust estimates to calculate parameters. The results for each one of the models contrasted are presented below.

One-factor model (one latent factor and 20 items as observed variables with their respective measurement errors): the analysis showed inadequate goodness-of-fit indices for this model (S-Bχ²/df = 13.40; PGFI = .89; CFI = .60; TLI = .59; RMSEA = .16, IC 90% [.15; .17]; AIC = 1216.42). Standardized factor loadings ranged from .37 to .78 (p < .001). The elimination of the three items with the lowest loadings (item 1 «¿Usted ha podido expresar sus puntos de vista y sentimientos durante la aplicación de los procedimientos?» [«Have you been able to express your views and feelings during those procedures?»]; item 2 «¿Usted ha tenido influencia sobre los logros obtenidos a partir de la aplicación de los procedimientos?» [«Have you had influence over the outcome arrived at by those procedures?»]; item 15 «¿Su jefe/supervisor/persona responsable evita hacer comentarios inapropiados? [«Has he/she refrained from improper remarks or comments?»]) did not produce any significant improvement in model fit (S-Bχ²/gl = 10.42; PGFI = .83; CFI = .61; TLI = .60; RMSEA = .15, IC 90% [.14; .16]; AIC = 1097.89). This one factor accounted for 41% of the variance of OJ.

Three-factor model (three interrelated factors and 20 items as observed variables with their respective measurement errors): the goodness-offit indices obtained for this model were below acceptable thresholds (S-Bχ²/df = 8.20; PGFI = .79; CFI = .78; TLI = .73; RMSEA = .12, IC 90% [.11; .13]; AIC = 769.39). The standardized factor loadings for the distributive and procedural justice factors showed identical values than those observed in the four-factor model. As for the interactional justice factor (which included items corresponding to both interpersonal and informational justice), saturations ranged from .41 to .86. The elimination of the two items with the lowest loadings (item 17 «¿Su jefe/ supervisor/persona responsable le explica con claridad los procedimientos adoptados?» [«Has he/she explained the procedures thoroughly?»]; item 19 «¿Su jefe/supervisor/persona responsable se comunica con usted en los momentos oportunos?» [«Has he/she communicated details in a timely manner?»]) did not yield substantial improvement in model fit (S-Bχ²/df = 7.31; PGFI = .77; CFI = .79; TLI = .78; RMSEA = .11, IC 90% [.10; .12]; AIC = 758.12). The three factors accounted for 60% of the variance of the construct.

Four-factor model (four correlated factors and 20 items as observed variables with their respective measurement errors): the indices showed a good fit to the data (S-Bχ2/df = 2.36; PGFI = .61; CFI = .94; TLI = .91; RMSEA = .03, IC 90% [.02; .04]; AIC = 401.64). Standardized factor loadings were significant in all items (p < .001). They ranged from .77 to .85 in the distributive justice factor, from .57 to .90 in the procedural justice factor, from .55 to .92 in the interpersonal justice factor, and from .69 to .84 in the informational justice factor. These four factors accounted for 67% of the variance of OJ.

In light of the criteria recommended by specialists (Kline, 2013), our results demonstrate that the most appropriate OJ model is the one that includes four interrelated dimensions. Table 4 presents the pattern coefficients (beta coefficients) and the structural coefficients corresponding to the items of the Argentine version of the OJ Scale.

As can be seen in Table 4, convergent factor loadings were markedly higher than the divergent loadings. Nonetheless, a certain tendency to factorial complexity is observed, since the structural loadings of some items were slightly higher than .30 (Graham et al., 2003).

Analyses of factorial invariance and group differences

The factor structure suggested by the factor analysis was examined according to participants’ gender and occupation. The invariance analysis indicated that the oblique four-factor measurement model presented configural invariance between men and women (CFI = .92; TLI = .90; RMSEA = .04) as well as between the different occupations of employees (CFI = .91; TLI = .90; RMSEA = .04). The addition of restrictions to the main and crossfactor loadings (CFI = .91; TLI = .91; RMSEA = .04), to the intercepts (CFI = .91; TLI = .91; RMSEA = .03), and to variances and covariances (CFI = .91; TLI = .91; RMSEA = .04) did not show increases outside the expected range (Cheung & Rensvold, 2002). These results demonstrate the measurement invariance of the instrument across the different groups. Similarly, the results of the Student’s t-tests did not show significant differences between men and women in any of the OJ dimensions: procedural justice (t_{(2; 404)} = .299, p = .764), distributive justice (t_{(2; 404)} = .434, p = .661), interpersonal justice (t_{(2; 404}) = - .015, p = .989), and informational justice (t_{(2; 404)} = .656, p = .514). Likewise, the one-way ANOVA did not show differences in the occupational activities: procedural justice (F _{(3; 402)} = 2.20, p = .098), distributive justice (F_{(3; 402)} = 2.41, p = .079), interpersonal justice (F _{(3; 402)} = 1.36, p = .241), and informational justice (F _{(3; 402)} = 1.86, p = .134).

Analyses of reliability and convergent/discriminant validity

After confirming the four-factor structure of the scale, as well as its measurement and configural invariance, reliability (ordinal alpha coefficients) and convergent/discriminant validity (AVE coefficients and their square roots) were examined.

As shown in Table 5, the four factors evidence adequate internal consistency and composite reliability. Although the correlations between the OJ factors are moderately high, AVE values, as well as their square roots, indicate that the four justice dimensions have adequate convergent and discriminant validity. In other words, each of the subscales measures a different facet of OJ without redundancy.

Discussion

The aim of this research paper was to provide evidence of the psychometric properties of the Argentine version of the OJ Scale developed by Colquitt (2001). Our findings indicate that the scale presents a fourfactor structure, acceptable convergent/discriminant validity, and satisfactory internal consistency. As regards the factor structure of the instrument, three alternative models were tested: (a) a model that comprised a latent factor with 20 items as observed variables, which was equivalent to the one identified by Blakely, Andrews, and Moorman (2005); (b) a model that consisted of three interrelated factors, one of which combined the social dimensions of justice (i.e. interpersonal and informational justice), and was analogous to the one reported by Spell and Arnold (2007); and (c) a model of four interrelated factors in line with the theoretical basis of the OJ construct operationalized by Colquitt (2001). Of the three models tested, the four-factor model showed the best fit to the data, indicating that distributive, procedural, interpersonal, and informational justice are four separate, though interrelated, dimensions of OJ. This conclusion is consistent not only with the structure and theoretical framework of the original scale (Colquitt, 2001), but also with the results of validation studies conducted in other socio-cultural and work contexts such as France, Germany, Italy, Japan, and Norway (Bergon, 2014; Di Fabio, 2008; Enoksen, 2015; Maier et al., 2007; Olsen et al., 2012; Shibaoka et al., 2010).

Our results also indicate that the Argentine version of the OJ Scale has good convergent/discriminant validity, meaning that the variance explained by the four factors is greater than the variance due to the measurement errors, and that the construct can be satisfactorily explained by the chosen indicators. The observed inter-factor correlations, although high, are in line with those reported by other instrumental studies. For example, in their Japanese validation of the scale, Shibaoka et al. (2010) observed correlations that ranged between .51 and .71. In the same way, both the Norwegian (Olsen et al., 2012) and the Spanish validation studies (Díaz-Gracia et al., 2014) found positive correlations between .45 and .72. Similar results were obtained by Rodríguez-Montalbán et al. (2015) in Puerto Rico, who reported correlations in the range of .53 to .74.

The findings of this study provide clear evidence of the theoretical postulates that support the multidimensionality of the OJ construct, which is integrated by the four dimensions of distributive, procedural, interpersonal, and informational justice (Colquitt, 2001; Colquitt et al., 2001; Colquitt et al., 2005; Colquitt & Shaw, 2005). Furthermore, the scale demonstrated good internal consistency, comparable to that of the original version and within the range informed by other studies (Enoksen, 2015; Shibaoka et al., 2010). The ordinal alpha coefficient for each dimension was greater than the coefficient for the total scale, which indicates that the different subscales could be used separately for practical purposes. This feature makes the OJ Scale a versatile and parsimonious tool that can meet the needs of researchers, managers, and/or human resource specialists.

Limitations and practical implications of the study

Although the convenience sampling method could be a possible limitation for the generalization of the results of the study, the complementary analyses performed showed strict and configural invariance in relation to employees’ gender and occupations. That is, no significant differences in OJ dimensions were observed between men and women or between the different branches of occupational activities. Another weakness could be linked to the external validity of the study. In this regard, since the stability of the dimensions has not been verified over time, future research should examine the test-retest reliability of the instrument. Finally, it should be noted that the scale explores perceived OJ, so participants’ responses could be contaminated by social desirability and/or by other subjective components, all of which could negatively impact on the generalizability of the results.

Despite these limitations, the availability of a scale with adequate psychometric properties to measure OJ perceptions has practical implications for both organizations and employees. From the organization’s point of view, the regular exploration of perceived justice could help implement organizational changes that create healthier and more productive work environments. For example, justice perceptions can be promoted among employees during instances associated with «winners» and «losers», such as personnel selection, the allocation of rewards and benefits, performance evaluations, conflict resolutions, and even dismissal processes. From the employee’s point of view, the regular measurement and monitoring of OJ can help increase well-being and satisfaction (Cassar & Buttigieg, 2015; Nery, Neiva, & Mendonça, 2016), improve performance and trust in the organization (Cheng, 2014), foster organizational citizenship behaviors, stimulate the generation of more altruistic and less retaliatory organizational climates (Colquitt et al., 2013), and prevent turnover intentions (Silva & Caetano, 2016). This pivotal role of OJ is well documented in the scientific literature, to the extent that perceived unfairness is associated with higher levels of stress and work absenteeism (Colquitt et al., 2013) and with a broad spectrum of counterproductive work behaviors (harassment, fraud, sabotage, theft) that employees execute in a failed attempt to restore the lost balance (Omar, Vaamonde, & Uribe Delgado, 2012; Proost et al., 2015).

Conclusions

Justice is like a two-sided coin. On the negative side, the lack of justice is associated with negative emotions, with an intensification of occupational stress responses, and with a greater motivation to engage in counterproductive behaviors towards the organization or its members. On the positive side, justice can do much more than prevent these regrettable results, as it acts as a buffer that allows employees to maintain respect and trust in their organization, and it is also related to effective leadership styles, higher commitment, greater job satisfaction, and increased motivation to perform organizational citizenship behaviors. Having a tool to measure perceptions of justice within organizations is not only the right thing to do, but also entails competitive advantages and business success. Hence the importance of this study that supports the four-factor structure of Colquitt’s OJ Scale, and suggests it is a valid and reliable measure to explore OJ perceptions in Argentine organizations.

Conflict of interest

The authors declare that they have no conflict of interest associated with this work.

Ethical responsibility

The authors declare that no experiments were carried out for this study and that all procedures involving human participants were in full accordance with the ethical standards established by the 1964 Declaration of Helsinki and its later amendments, and by the 2017 American Psychological Association Ethical Principles of Psychologists and Code of Conduct. Data confidentiality: The authors declare that the CONICET ethical recommendations for research in the social and human sciences (Resolution 2827/06) have been strictly followed to protect participants’ rights. Thus anonymity and confidentiality were assured throughout the research process. Right to privacy and informed consent: Personal data of participants were not required for the study. Informed consent was obtained from all individuals included in the study.

Funding

Financial support for this research was provided by the National Scientific and Technical Research Council (CONICET), Argentina. Grant No. 112-201101-00001.

 

Referencias

Ambrose, M. L., & Schminke, M. (2009). The role of overall justice judgments in organizational justice research: A test of mediation. Journal of Applied Psychology, 94, 491-500. doi: 10.1037/a0013203         [ Links ]

American Psychological Association (2017). Ethical principles of psychologists and code of conduct. Retrieved from http://www.apa.org/ethics/code/         [ Links ]

Ato, M., López, J. J., & Benavente, A. (2013). Un sistema de clasificación de los diseños de investigación en psicología [A classification system of research designs in psychology]. Anales de Psicología, 29, 1038-1059. doi: 10.6018/analesps.29.3.178511         [ Links ]

Bagozzi, R. P., & Yi, Y. (2012). Specification, evaluation, and interpretation of structural equation models. Journal of the Academy of Marketing Science, 40, 8-34. doi: 10.1007/s11747-011-0278-x         [ Links ]

Bentler, P. (2006). EQS 6 structural equations program manual. Encino: Multivariate Software. doi: 10.4324/9780203726532        [ Links ]

Bergon, G. (2014). Validation d’une échelle de mesure de la justice inter-organisationnelle dans un contexte «Donneur d’Ordre/Sous-Traitant». Revue Management et Avenir, 70, 171-187. doi: 10.3917/mav.070.0171        [ Links ]

Bies, R. J., & Moag, J. S. (1986). Interactional justice: Communication criteria for justice. In B. Sheppard (Ed.), Research on negotiation in organizations (Vol. 1, pp. 43-55). Greenwich, CT: JAI Press.         [ Links ]

Bies, R. J., & Shapiro, D. L. (1987). Interactional fairness judgments: The influence of causal accounts. Social Justice Research, 1, 199-218. doi: 10.1007/BF01048016         [ Links ]

Blakely, G., Andrews, M., & Moorman, R. (2005). The moderating effects of equity sensitivity on the relationship between organizational justice and organizational citizenship behaviors. Journal of Business and Psychology, 20, 259-273. doi: 10.1007/s10869-005-8263-3         [ Links ]

Cassar, V., & Buttigieg, S. (2015). Psychological contract breach, organizational justice and emotional wellbeing. Personnel Review, 44, 217-235. doi: 10.1108/PR-04-2013-0061         [ Links ]

Cheng, S. Y. (2014). The mediating role of organizational justice on the relationship between administrative performance appraisal practices and organizational commitment. The International Journal of Human Resources Management, 24(8), 1131-1148. doi: 10.1080/09585192.2013.816864        [ Links ]

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of- fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233-255. Retrieved from https://doi.org/10.1207/S15328007SEM0902         [ Links ]

Cho, Y., & Sai, N. (2013). Does organizational justice matter in the federal workplace? Review of Public Personnel Administration, 33, 227-251. doi: 10.1177/0734371x12458126         [ Links ]

Colquitt, J. (2001). On the dimensionality of organizational justice: A construct validation of a measure. Journal of Applied Psychology, 86, 386-400. doi: 10.1037/0021-9010.86.3.386         [ Links ]

Colquitt, J. A., & Shaw, J. C. (2005). How should organizational justice be measured? In J. Greenberg & J. C. Colquitt (Eds.), Handbook of Organizational Justice (pp. 113-151). Mahwah, NJ: Erlbaum.         [ Links ]

Colquitt, J. A., Greenberg, J., & Zapata-Phelan, S. P. (2005). What is organizational justice? A historical overview. In J. Greenberg & J. A. Colquitt (Eds.), Handbook of organizational justice (pp. 3-56). Mahwah, NJ: Erlbaum.         [ Links ]

Colquitt, J. A., Scott, B. A., Rodell, J. B., Long, D. M., Zapata, C. P., Conlon, D. E., & Wesson, M. J. (2013). Justice at the millennium, a decade later: A metaanalytic test of social exchange and affect-based perspectives. Journal of Applied Psychology, 98, 199-236. doi: 10.1037/a0031757         [ Links ]

Colquitt, J. A., Wesson, M. J., Porter, C. O. L. H., Conlon, D. E., & Ng, K. Y. (2001). Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology, 86(3), 425-445. doi: 10.1037/0021-9010.86.3.425         [ Links ]

Consejo Nacional de Investigaciones Científicas y Técnicas (2006). Resolución D No. 2857: CONICET: Lineamientos para el comportamiento ético en las Ciencias Sociales y Humanidades. Buenos Aires: CONICET.         [ Links ]

Cropanzano, R., Bowen, D., & Gilliland, S. (2007) The Management of Organizational Justice. Academic of Management Perspective, 21(4), 434-448. doi: 10.5465/amp.2007.27895338        [ Links ]

Di Fabio, A. (2008). Strumenti per il counseling organizzativo: Proprietà psicometriche della versione Italiana dell’Organizational Justice Scale (OJS). Counseling: Giornale Italiano di Ricerca e Applicazioni, 1(2), 209-223.         [ Links ]

Díaz-Gracia, L., Barbaranelli, C., & Moreno-Jiménez, B. (2014). Spanish version of Colquitt’s Organizational Justice Scale. Psicothema, 26, 538-544. doi: 10.7334/psicothema2014.110         [ Links ]

Elovainio, M., Heponiemi, T., Sinervo, T., & Magnavita, N. (2010). Organizational justice and health: Review of evidence. Giornale Italiano di Medicina del Lavoro ed Ergonomia, 32, B5-B9.         [ Links ]

Enoksen, E. (2015). Examining the dimensionality of Colquitt’s Organizational Justice Scale in a public health sector context. Psychological Reports, 116, 723-737. doi: 10.2466/01.PR0.116k26w0         [ Links ]

Gadermann, A. M., Guhn, M., & Zumbo, D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17(3), 1-13. Retrieved from http://pareonline.net/getvn.asp?v=17&n=3         [ Links ]

Goldman, B. M. (2003). The application of reference cognitions theory to legal-claiming by terminated workers: The role of organizational justice and anger. Journal of Management, 29, 705-728.         [ Links ]

Graham, J. M., Guthrie, A. C., & Thompson, B. (2003). Consequences of not interpreting structure coefficients in published CFA research: A reminder. Structural Equation Modeling, 10(1), 142-153. doi: 10.1207/S15328007SEM1001_7        [ Links ]

Greenberg, J. (1987). A taxonomy of organizational justice theories. Academy of Management Review, 12, 9-22. doi: 10.5465/amr.1987.4306437        [ Links ]

Greenberg, J. (1993). The social side of fairness: Interpersonal and informational classes of organizational justice. In R. Cropanzano (Ed.), Justice in the workplace: Approaching fairness in human resource management (pp. 79-103). Hillsdale, NJ: Erlbaum.         [ Links ]

Henseler, J., Ringle, C. M., & Sarstedt, M. J. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115-135. doi: 10.1007/s11747-014-0403-8         [ Links ]

Kline, R. B. (2013). Exploratory and confirmatory factor analysis. In Y. Petscher & C. Schatsschneider (Eds.), Applied quantitative analysis in the social sciences (pp. 171-207). New York, NY: Routledge.         [ Links ]

Leventhal, G. S. (1976). The distribution of rewards and resources in groups and organizations. In L. Berkowitz & W. Walster (Eds.), Advances in experimental social psychology (vol. 9, PP. 91-131). New York: Academic Press.         [ Links ]

Leventhal, G. S. (1980). What should be done with equity theory? New approaches to the study of justice in social relationships. In K. Gergen, M. Greenberg & R. Willis (Eds.), Social exchange: Advances in experimental and social psychology (pp. 91-131). New York, NY: Plenum. doi: 10.1007/978-1-4613-3087-5_2        [ Links ]

Lloret, S., Ferreres, A., Hernández, A., & Tomás, I. (2017). The exploratory factor analysis of items: Guided analysis based on empirical data and software. Anales de Psicología, 33(2), 417-432. doi: 10.6018/analesps.33.2.270211         [ Links ]

Maier, G. W., Streicher, B., Jonas, E., & Woschée, R. (2007). Assessment of justice in organizations: The validity of a German version of the questionnaire by Colquitt (2001). Diagnostica, 53, 97-108. doi: 10.1026/0012-1924.53.2.97        [ Links ]

Mladinic, A. (2002). Justicia organizacional: su impacto en las actitudes y conductas [Organizational justice: impact on attitudes and behaviors]. Unpublished manuscript, School of Psychology, Pontificia Universidad Católica de Chile, Santiago, Chile.         [ Links ]

Muñiz, J., Elosua, P., & Hambleton, R. (2013). Directrices para la traducción y adaptación de los tests: segunda edición. Psicothema, 25, 151-157. doi: 10.7334/psicothema2013.2         [ Links ]

Nasurdin, A. M., Ahmad, N. H., & Razalli, A. A. (2014). Politics, justice, stress, and deviant behaviour in organizations: An empirical analysis. International Journal of Business and Society, 15, 235-254. Retrieved from http://www.ijbs.unimas.my/repository/pdf/Vol15-no2-paper3.pdf         [ Links ]

Nery, V. F., Neiva, E. R., & Mendonça, H. (2016). The changing context and the organizational justice impact on the employee well-being. Paidéia (Ribeirão Preto), 26(65), 317-324. doi: 10.1590/1982-43272665201609         [ Links ]

Ohana, M. (2014). A multilevel study of the relationship between organizational justice and affective commitment: The moderating role of organizational size and tenure. Personnel Review, 43, 654-671. doi: 10.1108/pr-05-2013-0073         [ Links ]

Olsen, O. K., Myrseth, H., Eidhamar, A., & Hystad, S. W. (2012). Psychometric properties of a fourcomponent Norwegian Organizational Justice Scale. Psychological Reports, 110, 571-588. doi: 10.2466/01.08.14.PR0.110.2.571-588         [ Links ]

Omar, A., Oggero, H., Maltaneres, V., & Paris, L. (2003, March). Análisis de la estructura factorial de una escala para explorar justicia organizacional [Factorial structure analysis of a scale to explore organizational justice]. Paper presented at the 1st Marplatense Congress of Psychology, Mar de la Plata, Argentina.         [ Links ]

Omar, A., Vaamonde, J. D., & Uribe Delgado, H. (2012). Comportamientos contraproducentes en el trabajo: diseño y validación de una escala. Diversitas: Perspectivas en Psicología, 8(2), 249-265. doi: 10.15332/sl794-9998.2012.0002.04         [ Links ]

Proost, K., Verboon, P., & van Ruysseveldt, J. (2015). Organizational justice as buffer against stressful job demands. Journal of Managerial Psychology, 30, 487-499. doi: 10.1108/jmp-02-2013-0040         [ Links ]

Rodríguez-Montalbán, R., Martínez-Lugo, M., & Sánchez- Cardona, I. (2015). Análisis de las propiedades psicométricas de la escala de justicia organizacional de Colquitt en una muestra de empleados(as) en Puerto Rico. Revista Portorriqueña de Psicología, 26(2), 270-286. Retrieved from http://www.repsasppr.net/index.php/reps/article/view/298/277         [ Links ]

Schilpzand, M., Martins, L., Kirkman, B., Lowe, K., & Chen, Z. (2013). The relationship between organizational justice and organizational citizenship behaviour: The role of cultural value orientations. Management & Organization Review, 9, 345-374. doi: 10.1111/more.12014         [ Links ]

Shapiro, D. L., Buttner, E. H., & Barry, B. (1994). Explanations: What factors enhance their perceived adequacy? Organizational Behavior and Human Decision Processes, 58, 346-368. doi: 10.1006/obhd.1994.1041         [ Links ]

Shibaoka, M., Takada, M., Watanabe, M., Kojima, R., Kakinuma, M., Tanaka, K., & Kawakami, N. (2010). Development and validity of the Japanese version of the Organizational Justice Scale. Industrial Health, 48, 66-73. doi: 10.2486/indhealth.48.66         [ Links ]

Siegrist, J. (1996). Adverse health effects of high-effort/ low-reward conditions. Journal of Occupational Health Psychology, 1, 27. doi: 10.1037/1076-8998.1.1.27        [ Links ]

Silva, M. R., & Caetano, A. (2016). Organizational justice across cultures: A systematic review of four decades of research and some directions for the future. Social Justice Research, 29, 257-287. doi: 10.1007/S11211-016-0263-0        [ Links ]

Spell, C. S., & Arnold, T. J. (2007). A multi-level analysis of organizational justice climate, structure, and employee mental health. Journal of Management, 33, 724-751. doi: 10.1177/0149206307305560        [ Links ]

Streicher, B., Jonas, E., Maier, G. W., Frey, D., Woschée, R., & Wassmer, B. (2007). Test of the construct and criteria validity of a German measure of organizational justice. European Journal of Psychological Assessment, 24, 131-139. doi: 10.1027/1015-5759.24.2.131        [ Links ]

Sweeney, P. D., & McFarlin, D. B. (1997). Process and outcome: Gender differences in the assessment of justice. Journal of Organizational Behavior, 18, 83-93. doi: 10.1002/(SICI)1099-1379(199701)18:1<83::AID-JOB779>3.0.CO;2-3

Thibaut, J., & Walker, L. (1975). Procedural justice: A psychological analysis. Hillsdale, NJ: Erlbaum.         [ Links ]

 

---

Alicia Omar

Universidad Nacional de Rosario (Argentina)

National Scientific and Technical Research Council (CONICET), and National University of Rosario, Rosario, Argentina. Postal address: Italia 1365 Piso 1 Dpto. A, (2000), Rosario, Argentina

Tel. +54-341-4480314

Ph. D. in Psychology. She has conducted research on values, organizational culture, and constructs of positive organizational psychology. Her main area of research lies in organizational and health psychology.

ORCID: 0000-0001-6613-2565

Corresponding author: agraomar@yahoo.com

Solana Salessi

Universidad Nacional de Rafaela (Argentina)

National Scientific and Technical Research Council (CONICET), and National University of Rafaela, Santa Fe, Argentina. Ph. D. in Psychology. She has carried out research on job satisfaction, innovative behavior, and structural equation modeling. Her main research interests focus on organizational and educational psychology.

ORCID: 0000-0001-9496-9493

solanasalessi@gmail.com

Juan Diego Vaamonde

Universidad Nacional de Rafaela (Argentina)

National University of Rosario, Rosario, and National University of Rafaela, Santa Fe, Argentina Ph. D. in Psychology. He has conducted research on organizational justice, occupational stress, sexism, and turnover intentions. His main area of research is in organizational and health psychology.

ORCID: 0000-0002-2205-5075

juandvaamonde@yahoo.com.ar

Florencia Urteaga

Universidad Autónoma de Entre Ríos (Argentina)

National University of Entre Ríos, Paraná, and Autonomous University of Entre Ríos, Paraná, Argentina. Ph. D. in Engineering. She has done research on organizational culture and human resource management practices. Her main research interests focus on multivariate statistics, data mining, and quantitative modeling.

ORCID: 0000-0002-5469-8067

afurteaga@gmail.com

Para citar este artículo:

Omar, A., Salessi, S., Vaamonde, J., & Urqueaga, F. (2018). Psychometric properties of Colquitt’s Organizational Justice Scale in Argentine workers. Liberabit, 24(1), 61-79. doi: 10.24265/liberabit.2018.v24n1.05

 

Este es un artículo Open Access bajo la licencia Creative Common (Atribución-NoComercial-CompartirIgual 4.0)

 

Recibido: 11 de noviembre de 2017

Aceptado: 20 de mayo de 2018