The item analysis of the 2001 ASA exam is now complete. We counted 302 of the Saturday exam returns before we called a halt. The three sets of one hundred showed very similar results. A big thank you is due to our chief tabulator George Morgan, his computerized consultant Keith Wilcox and our graphics designer Rob Spencer.
Our returns from the Sabbath observer exam were also counted, but separately, since Personnel scrambles question order between the regular and the Sabbath observer exams.
The chart looks complex, but it is easy to use. For question #1 on the Saturday exam, all of the results are reported, but the answer chosen by the most candidates is in "bold" print. The most common answer is not necessarily the correct answer, but often it is. If 97% of the test takers picked a particular answer and you did not, you probably have good reason to worry.
For some questions, more than one answer was highlighted because no one answer was chosen by a majority of test takers. Instead, two different answers were each somewhat popular. The best example of this is question #71. Answers A and C each were chosen by 47% of the candidates.
Appeals. OSA's letter instructing candidates to request the right to protest was modified by the instructions given on the pink sheet handed out by the examiners.
As per the City's instructions, candidates had seven days to request a chance to review the test at some day in the future. Any member who missed the seven-day deadline can still send in a request and hope for the best.
The actual protest, original and four copies, does not have to be submitted until after July 9th, the fifth Monday after the exam. It is expected that the City answer key will be published by that date. Protests have to be postmarked no later than 30 days thereafter. The real question is how you can protest at all without a copy of the exam on hand. It is hard, but there are hundreds of candidates who will make the effort.
If you did not keep your answer key, you can only await the score card in the mail.
If you did keep your answer sheet but did not (or do not) ask to review the exam, you will have to base the protests on memory. Most of us cannot do this, but some can.
If you kept your answer sheet and did ask to review the exam, our analysis plus the City's list of correct answers will enable you to focus on protesting. You will protest the questions (a) that you got wrong and (b) that many others got "wrong" as well. On #71 almost everybody chose either A or C, and in such cases there is good reason to review the question carefully. Many times, after protests, Personnel has agreed that there are two correct answers to a badly written question.
The exam itself will be undergoing a validation process this month. We expect that our brothers and sisters at Personnel/DCAS will be using their data to do a thorough item analysis and will be checking the exam over very carefully. Once completed, the official answer key will be issued.
OSA regrets that the City no longer allows candidates to take their exams home. We believe the design is to actively discourage protests. On the other hand, we are assured by Joan Doheny, OSA Chapter Chairperson from DCAS/Personnel that the protests submitted are, in fact, carefully considered by the professional staff of the Agency.
Conclusions. The 2001 exam was administered quite well compared to earlier years. One school did start late, but only 45 minutes late, and the others started on time. The weather was nice for those on line before the exam. By afternoon, some classrooms were too warm but others, with a breeze, were comfortable.
High school seats are too small for our members and the desks are a trial, so in the future, we will ask DCAS to seek accommodations in a college.
In Canarsie, (the site of the famous football game outside the windows last exam), there was a car radio (alarm?) that distracted some candidates for a bit but the area was quiet thereafter.
The exam was not easy. There were 18 questions where the agreement on the right answer topped 90%, but there were also 19 questions where there was no majority choice at all. Put another way, on average, most contestants got those 19 questions wrong. OSA will be urging DCAS to add points to the raw scores as we have in the past. Our tool, as you would expect, will be the item analysis you have in hand. In the past, DCAS has modified the raw scores somewhat, so we will keep our hopes up.
ASSOCIATE STAFF ANALYST EXAM
Item Analysis – Saturday June 9, 2001 Exam – Based On 302 Answer Sheets
|
Question No. |
No Answer |
Answer "A" Raw No. |
Answer "A" Percent |
Answer "B" Raw No. |
Answer "B" Percent |
Answer "C" Raw No. |
Answer "C" Percent |
Answer "D" Raw No. |
Answer "D" Percent |
Most Common Answer |
|
1 |
30 |
10% |
10 |
3% |
182 |
60% |
80 |
26% |
C |
|
|
2 |
255 |
84% |
15 |
5% |
17 |
6% |
15 |
5% |
A |
|
|
3 |
144 |
48% |
16 |
5% |
123 |
41% |
19 |
6% |
A c |
|
|
4 |
6 |
2% |
48 |
16% |
159 |
53% |
89 |
29% |
C |
|
|
5 |
5 |
2% |
3 |
1% |
3 |
1% |
291 |
96% |
D |
|
|
6 |
7 |
2% |
269 |
89% |
8 |
3% |
18 |
6% |
B |
|
|
7 |
16 |
5% |
223 |
74% |
58 |
19% |
5 |
2% |
B |
|
|
8 |
26 |
9% |
228 |
75% |
27 |
9% |
21 |
7% |
B |
|
|
9 |
243 |
80% |
10 |
3% |
13 |
4% |
36 |
12% |
A |
|
|
10 |
19 |
6% |
24 |
8% |
180 |
60% |
79 |
26% |
C |
|
|
11 |
67 |
22% |
223 |
74% |
8 |
3% |
4 |
1% |
B |
|
|
12 |
251 |
83% |
20 |
7% |
6 |
2% |
25 |
8% |
A |
|
|
13 |
73 |
24% |
69 |
23% |
7 |
2% |
153 |
51% |
D |
|
|
14 |
0 |
0% |
19 |
6% |
4 |
1% |
279 |
92% |
D |
|
|
15 |
34 |
11% |
14 |
5% |
230 |
76% |
24 |
8% |
C |
|
|
16 |
1 |
36 |
12% |
8 |
3% |
24 |
8% |
233 |
77% |
D |
|
17 |
43 |
14% |
51 |
17% |
178 |
59% |
30 |
10% |
C |
|
|
18 |
9 |
3% |
248 |
82% |
30 |
10% |
15 |
5% |
B |
|
|
19 |
3 |
1% |
75 |
25% |
52 |
17% |
172 |
57% |
D |
|
|
20 |
1 |
11 |
4% |
39 |
13% |
21 |
7% |
230 |
76% |
D |
|
21 |
169 |
56% |
30 |
10% |
12 |
4% |
91 |
30% |
A |
|
|
22 |
10 |
3% |
274 |
91% |
4 |
1% |
14 |
5% |
B |
|
|
23 |
1 |
0% |
6 |
2% |
2 |
1% |
293 |
97% |
D |
|
|
24 |
9 |
3% |
276 |
91% |
4 |
1% |
13 |
4% |
B |
|
|
25 |
1 |
0% |
292 |
97% |
9 |
3% |
0 |
0% |
B |
|
|
26 |
16 |
5% |
19 |
6% |
13 |
4% |
253 |
84% |
D |
|
|
27 |
36 |
12% |
6 |
2% |
17 |
6% |
243 |
80% |
D |
|
|
28 |
39 |
13% |
89 |
29% |
29 |
10% |
145 |
48% |
D |
|
|
29 |
232 |
77% |
29 |
10% |
19 |
6% |
22 |
7% |
A |
|
|
30 |
11 |
4% |
199 |
66% |
34 |
11% |
58 |
19% |
B |
|
|
31 |
1 |
16 |
5% |
4 |
1% |
10 |
3% |
271 |
90% |
D |
|
32 |
197 |
65% |
14 |
5% |
58 |
19% |
33 |
11% |
A |
|
|
33 |
24 |
8% |
70 |
23% |
178 |
59% |
30 |
10% |
C |
|
|
34 |
29 |
10% |
87 |
29% |
172 |
57% |
14 |
5% |
C |
|
|
35 |
119 |
39% |
118 |
39% |
43 |
14% |
22 |
7% |
A/B |
|
|
36 |
1 |
140 |
46% |
79 |
26% |
44 |
15% |
38 |
13% |
A |
|
37 |
44 |
15% |
191 |
63% |
64 |
21% |
3 |
1% |
B |
|
|
38 |
9 |
3% |
74 |
25% |
80 |
26% |
139 |
46% |
D |
|
|
39 |
1 |
46 |
15% |
69 |
23% |
153 |
51% |
33 |
11% |
C |
|
40 |
1 |
54 |
18% |
149 |
49% |
38 |
13% |
60 |
20% |
B |
|
41 |
5 |
2% |
280 |
93% |
11 |
4% |
6 |
2% |
B |
|
|
42 |
283 |
94% |
16 |
5% |
3 |
1% |
0 |
0% |
A |
|
|
43 |
2 |
1% |
7 |
2% |
256 |
85% |
37 |
12% |
C |
|
|
44 |
0 |
0% |
204 |
68% |
7 |
2% |
91 |
30% |
B |
|
|
45 |
275 |
91% |
20 |
7% |
1 |
0% |
6 |
2% |
A |
|
|
46 |
52 |
17% |
16 |
5% |
9 |
3% |
225 |
75% |
D |
|
|
47 |
5 |
2% |
4 |
1% |
288 |
95% |
5 |
2% |
C |
|
|
48 |
2 |
1% |
4 |
1% |
12 |
4% |
284 |
94% |
D |
|
|
49 |
6 |
2% |
13 |
4% |
283 |
94% |
0 |
0% |
C |
|
|
50 |
251 |
83% |
26 |
9% |
10 |
3% |
15 |
5% |
A |
|
|
51 |
6 |
2% |
293 |
97% |
3 |
1% |
0 |
0% |
B |
|
|
52 |
52 |
17% |
232 |
77% |
2 |
1% |
16 |
5% |
B |
|
|
53 |
4 |
138 |
46% |
99 |
33% |
28 |
9% |
33 |
11% |
A b |
|
54 |
94 |
31% |
67 |
22% |
32 |
11% |
109 |
36% |
D a |
|
|
55 |
25 |
8% |
133 |
44% |
30 |
10% |
114 |
38% |
B d |
|
|
56 |
2 |
116 |
38% |
95 |
31% |
39 |
13% |
50 |
17% |
A b |
|
57 |
1 |
54 |
18% |
20 |
7% |
125 |
41% |
102 |
34% |
C d |
|
58 |
52 |
17% |
43 |
14% |
11 |
4% |
196 |
65% |
D |
|
|
59 |
149 |
49% |
23 |
8% |
34 |
11% |
96 |
32% |
A |
|
|
60 |
1 |
16 |
5% |
23 |
8% |
241 |
80% |
21 |
7% |
C |
|
61 |
36 |
12% |
215 |
71% |
14 |
5% |
37 |
12% |
B |
|
|
62 |
12 |
4% |
27 |
9% |
214 |
71% |
49 |
16% |
C |
|
|
63 |
28 |
9% |
31 |
10% |
38 |
13% |
205 |
68% |
D |
|
|
64 |
22 |
7% |
259 |
86% |
11 |
4% |
10 |
3% |
B |
|
|
65 |
190 |
63% |
82 |
27% |
21 |
7% |
9 |
3% |
A |
|
|
66 |
19 |
6% |
10 |
3% |
241 |
80% |
32 |
11% |
C |
|
|
67 |
8 |
3% |
6 |
2% |
278 |
92% |
10 |
3% |
C |
|
|
68 |
1 |
43 |
14% |
34 |
11% |
77 |
25% |
147 |
49% |
D |
|
69 |
33 |
11% |
7 |
2% |
21 |
7% |
241 |
80% |
D |
|
|
70 |
18 |
6% |
94 |
31% |
185 |
61% |
5 |
2% |
C |
|
|
71 |
142 |
47% |
14 |
5% |
143 |
47% |
3 |
1% |
C/A |
|
|
72 |
1 |
223 |
74% |
53 |
18% |
17 |
6% |
8 |
3% |
A |
|
73 |
15 |
5% |
8 |
3% |
272 |
90% |
7 |
2% |
C |
|
|
74 |
262 |
87% |
8 |
3% |
22 |
7% |
10 |
3% |
A |
|
|
75 |
20 |
7% |
30 |
10% |
49 |
16% |
203 |
67% |
D |
|
|
76 |
24 |
8% |
27 |
9% |
220 |
73% |
31 |
10% |
C |
|
|
77 |
26 |
9% |
35 |
12% |
228 |
75% |
13 |
4% |
C |
|
|
78 |
93 |
31% |
38 |
13% |
124 |
41% |
47 |
16% |
C a |
|
|
79 |
1 |
75 |
25% |
122 |
40% |
69 |
23% |
35 |
12% |
B |
|
80 |
1 |
46 |
15% |
79 |
26% |
108 |
36% |
68 |
23% |
C |
|
81 |
68 |
23% |
139 |
46% |
23 |
8% |
72 |
24% |
B |
|
|
82 |
99 |
33% |
22 |
7% |
168 |
56% |
13 |
4% |
C |
|
|
83 |
220 |
73% |
11 |
4% |
28 |
9% |
43 |
14% |
A |
|
|
84 |
248 |
82% |
20 |
7% |
16 |
5% |
18 |
6% |
A |
|
|
85 |
164 |
54% |
42 |
14% |
15 |
5% |
81 |
27% |
A |
|
|
86 |
33 |
11% |
229 |
76% |
25 |
8% |
15 |
5% |
B |
|
|
87 |
4 |
1% |
9 |
3% |
256 |
85% |
33 |
11% |
C |
|
|
88 |
276 |
91% |
8 |
3% |
12 |
4% |
6 |
2% |
A |
|
|
89 |
270 |
89% |
26 |
9% |
3 |
1% |
3 |
1% |
A |
|
|
90 |
1 |
52 |
17% |
92 |
30% |
139 |
46% |
18 |
6% |
C |
ASSOCIATE STAFF ANALYST EXAM
Item Analysis – Friday June 8, 2001 Exam – Sabbath Observers – Based On 9 Answer Sheets
|
Question No. |
Answer "A" |
Answer "B" |
Answer "C" |
Answer "D" |
Most Common Answer |
|
1 |
9 |
A |
|||
|
2 |
9 |
A |
|||
|
3 |
4 |
4 |
1 |
B/C |
|
|
4 |
1 |
7 |
1 |
C |
|
|
5 |
8 |
1 |
A |
||
|
6 |
3 |
2 |
4 |
C a |
|
|
7 |
1 |
2 |
6 |
C |
|
|
8 |
7 |
1 |
1 |
A |
|
|
9 |
1 |
3 |
5 |
D |
|
|
10 |
9 |
D |
|||
|
11 |
3 |
6 |
C |
||
|
12 |
1 |
7 |
1 |
B |
|
|
13 |
8 |
1 |
A |
||
|
14 |
8 |
1 |
C |
||
|
15 |
3 |
6 |
B |
||
|
16 |
1 |
7 |
1 |
B |
|
|
17 |
3 |
6 |
D |
||
|
18 |
1 |
2 |
6 |
D |
|
|
19 |
2 |
1 |
6 |
D |
|
|
20 |
3 |
1 |
3 |
2 |
A/C |
|
21 |
8 |
1 |
B |
||
|
22 |
9 |
D |
|||
|
23 |
7 |
2 |
B |
||
|
24 |
8 |
1 |
B |
||
|
25 |
1 |
1 |
7 |
D |
|
|
26 |
1 |
8 |
D |
||
|
27 |
6 |
3 |
A |
||
|
28 |
1 |
7 |
1 |
C |
|
|
29 |
1 |
3 |
5 |
C |
|
|
30 |
3 |
6 |
B |
||
|
31 |
2 |
4 |
1 |
2 |
B |
|
32 |
4 |
4 |
1 |
A/B |
|
|
33 |
2 |
1 |
6 |
D |
|
|
34 |
4 |
2 |
3 |
A c |
|
|
35 |
1 |
5 |
3 |
B |
|
|
36 |
3 |
2 |
4 |
D a |
|
|
37 |
8 |
1 |
A |
||
|
38 |
1 |
7 |
1 |
B |
|
|
39 |
1 |
8 |
D |
||
|
40 |
6 |
3 |
A |
||
|
41 |
9 |
D |
|||
|
42 |
1 |
8 |
C |
||
|
43 |
7 |
1 |
1 |
A |
|
|
44 |
1 |
8 |
B |
||
|
45 |
3 |
6 |
B |
||
|
46 |
3 |
6 |
B |
||
|
47 |
4 |
1 |
4 |
A/D |
|
|
48 |
1 |
5 |
3 |
B |
|
|
49 |
9 |
B |
|||
|
50 |
8 |
1 |
A |
||
|
51 |
1 |
8 |
C |
||
|
52 |
7 |
2 |
B |
||
|
53 |
9 |
A |
|||
|
54 |
1 |
8 |
D |
||
|
55 |
1 |
8 |
C |
||
|
56 |
1 |
8 |
D |
||
|
57 |
9 |
B |
|||
|
58 |
5 |
4 |
B c |
||
|
59 |
4 |
1 |
2 |
2 |
A |
|
60 |
2 |
3 |
2 |
2 |
B acd |
|
61 |
1 |
2 |
6 |
D |
|
|
62 |
5 |
4 |
A d |
||
|
63 |
8 |
1 |
C |
||
|
64 |
5 |
3 |
1 |
A |
|
|
65 |
7 |
2 |
C |
||
|
66 |
2 |
7 |
D |
||
|
67 |
1 |
8 |
B |
||
|
68 |
7 |
1 |
1 |
A |
|
|
69 |
1 |
8 |
C |
||
|
70 |
9 |
C |
|||
|
71 |
1 |
1 |
1 |
6 |
D |
|
72 |
1 |
8 |
D |
||
|
73 |
1 |
3 |
5 |
C |
|
|
74 |
4 |
5 |
C a |
||
|
75 |
5 |
3 |
1 |
A |
|
|
76 |
9 |
C |
|||
|
77 |
8 |
1 |
A |
||
|
78 |
3 |
6 |
D |
||
|
79 |
1 |
8 |
C |
||
|
80 |
3 |
5 |
1 |
C |
|
|
81 |
4 |
1 |
3 |
1 |
A c |
|
82 |
2 |
1 |
3 |
3 |
C/D |
|
83 |
4 |
3 |
2 |
A b |
|
|
84 |
8 |
1 |
B |
||
|
85 |
8 |
1 |
C |
||
|
86 |
7 |
2 |
A |
||
|
87 |
8 |
1 |
A |
||
|
88 |
5 |
1 |
1 |
2 |
A |
|
89 |
9 |
B |
|||
|
90 |
9 |
C |
