Different Types of Analyses Discussion
1. Sheet V#1 scores (includes all the teachers who used version 1 texts both native and
non-natives) with a total of 22
participants.
2. Sheet V#2 scores includes all the teachers who used version 2 texts both native and
non-natives)
with a total of 22 participants.
3. Sheet Natives_V1 (all natives who used version 1 texts) with a total of 11
participants.
4. Sheet Non_natives_V1 (all non-natives who used version 1 texts) with a total of 11
participants.
5. Sheet Natives_V2 (all natives who used version 2 texts) with a total of 11
participants.
6. Sheet Non_Natives_V2 (all non-natives who used version 2 texts) with a total of 11
participants.
7. Sheet Natives_V1&2 includes all the natives together who used both version 1 and 2
with a total of 22 participants.
8. Sheet Non-Natives_V1&2 includes all the natives together who used both version 1
and 2 with a total of 22 participants.
9. Here, I have combined all the data for natives and non-natives for different
types of analyses.
I need to identify, analyze and test the inter-rater reliability of the teachers in the different
groups as explained above and the intra-rater reliability of each rater. I also need to identify
the leniency and harshness of the raters as well. I also want explanations for each test
explaining the results.
For the inter-rater reliability I need to measure the following:
? I need to calculate the percent agreement (measure the agreement of the multiple
raters)
? I also need to use Cohen’s and Fleiss’s Kappa statistics. Cohen’s Kappa when we
compare between the two ratings (paper-based and computer-based) and the ones
between natives and non-natives. Fleiss’s Kappa to compare between the multiple
texts and raters.
? I also need to use the Kendall’s Coefficient (Kendall’s W) to measure the strength
and the relationship between the multiple raters.
For intra-rater reliability, we need to treat the data once as a single index for the whole
raters and as individual raters.
? We can use Cohen’s kappa and the correlation coefficient (we have each participant
rated three texts twice, once on paper-based and on computer). The intra-class-
correlation (ICC).
? I also need to measure the leniency and harshness of the groups based on the
above divisions of sheets, where I also need graphs. I need the analysis of severity
and leniency to be shown between natives vs non natives in both versions and ways
of ratings.
Also, I need all the data files for all the analyses. Also, I want to know which softwares will
you use.
In summary, I need the following analyses:
Inter-rater:
? Cohen’s Kappa inter-rater
? Fleiss’s Kappa inter-rater
? Kendall’s Coefficient (Kendall’s W)
Intra-rater:
? Cohen’s Kappa intra-rater
? Intra-class correlation test (ICC)
? Multi-faceted rasch model (to analyze leniency and harshness)
I am working on an assignment in which we asked the teachers to rate 3 texts twice. First
they rated the three of them on computer and then the same teachers rated them on paper.
We have got two versions of texts, version 1 and versions 2. So a group of teachers rated
version 1 (which have 3 texts) and another group of teacher rated version 2 (which also have
3 texts).
We have got English native teachers and non native teachers. Group 1 of teachers, in which
there were both native and non native teachers, rated the version 1 of the texts. Group 2 of
teachers, which also have got both native and non native teachers, rated the version 2 of
texts.
The data is attached in the excel file. The following is the description of how I organised the
different data sheets in excel:
1. Sheet V#1 scores (includes all the teachers who used version 1 texts both native and
non-natives) with a total of 22 participants.
2. Sheet V#2 scores includes all the teachers who used version 2 texts both native and
non-natives) with a total of 22 participants.
3. Sheet Natives_V1 (all natives who used version 1 texts) with a total of 11
participants.
4. Sheet Non_natives_V1 (all non-natives who used version 1 texts) with a total of 11
participants.
5. Sheet Natives_V2 (all natives who used version 2 texts) with a total of 11
participants.
6. Sheet Non_Natives_V2 (all non-natives who used version 2 texts) with a total of 11
participants.
7. Sheet Natives_V1&2 includes all the natives together who used both version 1 and 2
with a total of 22 participants.
8. Sheet Non-Natives_V1&2 includes all the natives together who used both version 1
and 2 with a total of 22 participants.
9. Here, I have combined all the data for natives and non-natives for different
types of analyses.
I need to identify, analyze and test the inter-rater reliability of the teachers in the different
groups as explained above and the intra-rater reliability of each rater. I also need to identify
the leniency and harshness of the raters as well. I also want explanations for each test
explaining the results.
For the inter-rater reliability I need to measure the following:
●
●
I need to calculate the percent agreement (measure the agreement of the multiple
raters)
I also need to use Cohen’s and Fleiss’s Kappa statistics. Cohen’s Kappa when we
compare between the two ratings (paper-based and computer-based) and the ones
between natives and non-natives. Fleiss’s Kappa to compare between the multiple
texts and raters.
●
I also need to use the Kendall’s Coefficient (Kendall’s W) to measure the strength
and the relationship between the multiple raters.
For intra-rater reliability, we need to treat the data once as a single index for the whole
raters and as individual raters.
●
●
We can use Cohen’s kappa and the correlation coefficient (we have each participant
rated three texts twice, once on paper-based and on computer). The intra-classcorrelation (ICC).
I also need to measure the leniency and harshness of the groups based on the
above divisions of sheets, where I also need graphs. I need the analysis of severity
and leniency to be shown between natives vs non natives in both versions and ways
of ratings.
Also, I need all the data files for all the analyses. Also, I want to know which softwares will
you use.
In summary, I need the following analyses:
Inter-rater:
Cohen’s Kappa inter-rater
Fleiss’s Kappa inter-rater
Kendall’s Coefficient (Kendall’s W)
Intra-rater:
Cohen’s Kappa intra-rater
Intra-class correlation test (ICC)
Multi-faceted rasch model (to test leniency and harshness)
Participants
Text Number
Paper-based scores
computer scores
P10
P10
P10
P4
P4
P4
P5
P5
P5
P1
P1
P1
P14
P14
P14
P16
P16
P16
p18
p18
p18
p20
p20
p20
p21
p21
p21
P23
P23
P23
P25
P25
P25
P27
P27
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
8
6
6
7
7
6
8
7
8
7
6
7
7
5
4
8
7
6
6
4
5
7
7
6
9
5
4
9
7
6
10
7
9
8
4
7
6
7
8
6
7
8
7
8
8
8
7
7
7
6
9
8
8
7
5
4
7
6
6
9
4
5
9
7
7
10
7
8
9
6
P27
P29
P29
P29
p31
p31
p31
P32
P32
P32
P35
P35
P35
P37
P37
P37
P39
P39
P39
P41
P41
P41
P44
P44
P44
P46
P46
P46
P48
P48
P48
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
4
6
5
6
9
6
9
10
7
5
9
6
7
7
7
6
9
8
7
8
7
7
10
10
9
7
6
5
4
7
7
3
6
6
7
9
8
8
10
7
10
8
8
6
7
8
5
8
9
9
10
7
7
10
10
9
7
7
7
10
8
8
Participants
P13
P13
P13
P15
P15
P15
P6
P6
P6
P12
P12
P12
P11
P11
P11
P2
P2
P2
p9
p9
p9
p19
p19
p19
P22
P22
P22
P24
P24
P24
P26
P26
P26
P28
P28
P28
p30
p30
p30
P33
P33
P33
P34
P34
P34
P36
P36
P36
Text Number
Paper-based scores
computer scores
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
5
7
5
5
7
5
6
8
5
6
9
6
7
7
5
6
6
5
7
7
5
6
7
4
6
7
6
9
7
5
7
6
3
8
9
10
6
5
3
6
8
5
7
7
5
5
6
5
7
7
5
5
6
6
5
6
5
6
9
6
8
6
4
6
7
6
6
8
6
5
8
5
7
7
6
8
5
6
5
6
4
9
9
8
7
6
3
6
7
6
7
7
6
4
6
5
P38
P38
P38
P40
P40
P40
P43
P43
P43
P45
P45
P45
P47
P47
P47
P49
P49
P49
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
7
9
5
7
6
5
4
6
4
8
7
6
7
8
6
6
9
7
4
6
5
8
8
6
4
7
5
6
8
4
7
7
5
6
7
9
Participants
Text Number
Paper-based scores
computer scores
P10
P10
P10
P4
P4
P4
P1
P1
P1
P14
P14
P14
P32
P32
P32
P35
P35
P35
P37
P37
P37
P39
P39
P39
P41
P41
P41
P46
P46
P46
P48
P48
P48
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
8
6
6
7
7
6
7
6
7
7
5
4
10
7
5
9
6
7
7
7
6
9
8
7
8
7
7
7
6
5
4
7
7
7
6
7
8
6
7
8
8
7
7
7
6
10
7
10
8
8
6
7
8
5
8
9
9
10
7
7
7
7
7
10
8
8
Participants
P5
P5
P5
P16
P16
P16
p18
p18
p18
p20
p20
p20
p21
p21
p21
P23
P23
P23
P25
P25
P25
P27
P27
P27
P29
P29
P29
p31
p31
p31
P44
P44
P44
Text NumberPaper-based scorescomputer scores
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
8
7
8
8
7
6
6
4
5
7
7
6
9
5
4
9
7
6
10
7
9
8
4
4
6
5
9
6
9
10
10
9
8
7
8
9
8
8
7
5
4
7
6
6
9
4
5
9
7
7
10
7
8
9
6
3
6
6
7
9
8
8
10
10
9
Participants
P13
P13
P13
P15
P15
P15
P6
P6
P6
p9
p9
p9
P33
P33
P33
P36
P36
P36
P38
P38
P38
P40
P40
P40
P43
P43
P43
P47
P47
P47
P49
P49
P49
Text Number
Paper-based scores
computer scores
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
5
7
5
5
7
5
6
8
5
7
7
5
6
8
5
5
6
5
7
9
5
7
6
5
4
6
4
7
8
6
6
9
7
7
7
5
5
6
6
5
6
5
6
8
6
6
7
6
4
6
5
4
6
5
8
8
6
4
7
5
7
7
5
6
7
9
Participants
P12
P12
P12
P11
P11
P11
P2
P2
P2
p19
p19
p19
P22
P22
P22
P24
P24
P24
P26
P26
P26
P28
P28
P28
p30
p30
p30
P34
P34
P34
P45
P45
P45
Text Number
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
Paper-based scores computer scores
6
9
6
7
7
5
6
6
5
6
7
4
6
7
6
9
7
5
7
6
3
8
9
10
6
5
3
7
7
5
8
7
6
6
9
6
8
6
4
6
7
6
5
8
5
7
7
6
8
5
6
5
6
4
9
9
8
7
6
3
7
7
6
6
8
4
Participants
P10
P10
P10
P4
P4
P4
P1
P1
P1
P14
P14
P14
P32
P32
P32
P35
P35
P35
P37
P37
P37
P39
P39
P39
P41
P41
P41
P46
P46
P46
P48
P48
P48
P13
P13
P13
P15
P15
P15
P6
P6
P6
p9
p9
p9
Text NumberPaper-based scores
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
4
8
24
4
8
24
4
8
24
4
8
24
8
6
6
7
7
6
7
6
7
7
5
4
10
7
5
9
6
7
7
7
6
9
8
7
8
7
7
7
6
5
4
7
7
5
7
5
5
7
5
6
8
5
7
7
5
7
6
7
8
6
7
8
8
7
7
7
6
10
7
10
8
8
6
7
8
5
8
9
9
10
7
7
7
7
7
10
8
8
7
7
5
5
6
6
5
6
5
6
8
6
P33
P33
P33
P36
P36
P36
P38
P38
P38
P40
P40
P40
P43
P43
P43
P47
P47
P47
P49
P49
P49
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
6
8
5
5
6
5
7
9
5
7
6
5
4
6
4
7
8
6
6
9
7
6
7
6
4
6
5
4
6
5
8
8
6
4
7
5
7
7
5
6
7
9
Participants
P5
P5
P5
P16
P16
P16
p18
p18
p18
p20
p20
p20
p21
p21
p21
P23
P23
P23
P25
P25
P25
P27
P27
P27
P29
P29
P29
p31
p31
p31
P44
P44
P44
P12
P12
P12
P11
P11
P11
P2
P2
P2
p19
p19
p19
Text NumberPaper-based scores
computer-based scores
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
4
8
24
4
8
24
4
8
24
4
8
24
8
7
8
8
7
6
6
4
5
7
7
6
9
5
4
9
7
6
10
7
9
8
4
4
6
5
9
6
9
10
10
9
6
9
6
7
7
5
6
6
5
6
7
4
8
7
8
9
8
8
7
5
4
7
6
6
9
4
5
9
7
7
10
7
8
9
6
3
6
6
7
9
8
8
10
10
9
6
9
6
8
6
4
6
7
6
5
8
5
P22
P22
P22
P24
P24
P24
P26
P26
P26
P28
P28
P28
p30
p30
p30
P34
P34
P34
P45
P45
P45
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
6
7
6
9
7
5
7
6
3
8
9
10
6
5
3
7
7
5
8
7
6
7
7
6
8
5
6
5
6
4
9
9
8
7
6
3
7
7
6
6
8
4
Participants
Group
version
Text Number
Paper-based scores
P10
P10
P10
P4
P4
P4
P1
P1
P1
P14
P14
P14
P32
P32
P32
P35
P35
P35
P37
P37
P37
P39
P39
P39
P41
P41
P41
P46
P46
P46
P48
P48
P48
P13
P13
P13
P15
P15
P15
P6
P6
P6
p9
p9
p9
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
4
8
24
4
8
24
4
8
24
4
8
24
8
6
6
8
7
7
7
6
7
7
5
4
10
7
5
9
6
7
7
7
6
9
8
7
8
7
7
7
6
5
4
7
7
5
7
5
5
7
5
6
8
5
7
7
5
P33
P33
P33
P36
P36
P36
P38
P38
P38
P40
P40
P40
P43
P43
P43
P47
P47
P47
P49
P49
P49
P5
P5
P5
P16
P16
P16
p18
p18
p18
p20
p20
p20
p21
p21
p21
P23
P23
P23
P25
P25
P25
P27
P27
P27
P29
P29
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
12
1
5
6
8
5
5
6
5
7
9
5
7
6
5
4
6
4
7
8
6
6
9
7
8
7
8
8
7
6
6
4
5
7
7
6
9
5
4
9
7
6
10
7
9
8
4
4
6
5
P29
p31
p31
p31
P44
P44
P44
P12
P12
P12
P11
P11
P11
P2
P2
P2
p19
p19
p19
P22
P22
P22
P24
P24
P24
P26
P26
P26
P28
P28
P28
p30
p30
p30
P34
P34
P34
P45
P45
P45
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
Non-Native
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
12
1
5
12
1
5
12
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
4
8
24
6
9
6
9
10
10
9
6
9
6
7
7
5
6
6
5
6
7
4
6
7
6
9
7
5
7
6
3
8
9
10
6
5
3
7
7
5
8
7
6
Computer-based
7
6
7
8
6
7
8
8
7
7
7
6
10
7
10
8
8
6
7
8
5
8
9
9
10
7
7
7
7
7
10
8
8
7
7
5
5
6
6
5
6
5
6
8
6
6
7
6
4
6
5
4
6
5
8
8
6
4
7
5
7
7
5
6
7
9
8
7
8
9
8
8
7
5
4
7
6
6
9
4
5
9
7
7
10
7
8
9
6
3
6
6
7
9
8
8
10
10
9
6
9
6
8
6
4
6
7
6
5
8
5
7
7
6
8
5
6
5
6
4
9
9
8
7
6
3
7
7
6
6
8
4