March 6, 2003

 To:                   Dr. Jo Lynne DeMary, Superintendent of Public Instruction

Dr. Yvonne Brandon, Richmond Public School PASS Contact

 From:               James Heywood, Ed.D., Director

Office of School Improvement

 Subject:           The Need to Bring Richmond City Nine-Weeks’ Tests into Compliance with the PASS Agreement

 Problem

The use of Litespan/Edutest as a nine-weeks’ test measuring student and program progress on the SOL produces unreliable results. The degree of the problem varies from school to school and is partially related to the frequency of use of the Litespan/Edutest item pool by the classroom teachers prior to the test. 

The problem became apparent in examining the quarterly report for Woodville. In the first nine-weeks’, 47% of the students failed the English test using Flanagan in grades 1 and 2 and Edutest in grades 2-5. In the second quarter, only 23% failed using the same test combination, but 41% failed using Flanagan for all grades. The Edutest items for the first quarter were not contaminated since the Edutest program was not available at that time, so that 47% number may be reliable. During the second quarter, the staff readily admitted aggressively using the Edutest items, as encouraged, in the classroom.  Given the over-use of the items, the 23% score was not an accurate reflection of student knowledge at that time.

 The PASS Memorandum of Agreement specifies that the school division will “ensure the security of the nine-weeks’ test item data base by protecting the items used in the nine-weeks’ test from access and use by teachers on unit tests.” (page 4, Memorandum of Agreement, Model II)

 Causes

 ·            The math and English supervisors make up the Litespan/Edutest  and use the same item pool as is available to the teachers, allegedly because the pool is so small. This contaminates the items.  Some SOL have as few as five items.

Dr. Jo Lynne DeMary
Dr. Yvonne Brandon
Page 2

March 6, 2003

 ·            RPS encourages and tracks staff use of the Edutest items for classroom use. The result is, the more they are used, the more they are contaminated for use on the nine-weeks’ tests.

 ·            There are concerns with the alignment of Litespan/Edutest in some subject areas. Schools that have given both Flanagan and Litespan/Edutest for the same quarter find significantly different results in some subjects. In addition, PASS partner teams have commented on poor alignment and format of some items on Litespan/Edutest.

 ·            Woodville staff have remarked there are 50 items available in the pool (assumedly for a grade level) and they have heard students remark they "remember" the item on the nine-weeks’ test from classroom tests.

 ·            The problem will only get worse for the third quarter as the pool is continually accessed.

 Recommendations

 ·            Move into compliance with the PASS Memorandum of Agreement.

 ·            For the remainder of 2002-03, use Flanagan for the nine-weeks’ English and mathematics tests. Do not allow any teacher or school access to the Flanagan items prior to the nine-weeks’ test.

 ·            Continue to use Litespan/Edutest to enable teachers to use for informal and classroom assessment. Discontinue as a nine-weeks’ test.

 ·            Ask Litespan/Edutest to increase the item pool and verify the alignment in format and content, in collaboration with RPS staff, who can highlight the problem areas. If this is not done, an option would be to review the continuation of the Litespan/Edutest three-year contract.

 ·            For 2003-04, use the forthcoming SOLAR software engine or an equivalent and the Hampton-Region II consortium SOL-correlated database for the nine-weeks’ tests in English and Math.

 JSH/pl