Parts I and II of the UltraScore ™ API AYP Score Improvement Study, the results overview and alerts, require that we follow every yearly rule change, that we attend all the meetings and conferences that administrators would like to attend if they had the time, and that we apply this new yearly information to fine-tune the study and issue correct critical alerts to the school administrator. 

It is Part III of your UltraScore ™ study, the ULTRASCORE Intelligent Remediation Lists, that is revolutionary.   If a company or agency could just decipher all the cluster information from the STAR results file and translate this information into lists of precise weaknesses for each student, that would be a great aid to helping the student and the school.  In 1995-1997, that’s exactly what we did --- and many school districts thought our work was an enormous step forward.  Our work was originally marketed as the Student Achievement Study and then as the SAT9 Student Improvement Study.

But we examined the statistical results of our work --- and we weren’t satisfied.  In those years, there was no API. There was no NCLB. There was no AYP.  There was no AMO .  The California Department of Education held schools to a single standard.  They wanted schools to have half their students above the 50^th National Percentile in TotalReading and TotalMath.  Our budding system was able to provide an average increase of 8.1% .  This was better than anything offered by any other company --- but we felt we could do much better. 

In 1997-1999, however, we brought together two things to form the basis for the UltraScore ™ as it exists today.  First, the California Department of Education transformed its rather vague set of Frameworks for each subject into a set of Content Standards.  This had far-reaching implications for education and accountability --- but also for remediation systems such as ours.  Second, we at EdTech Associates realized that we were sitting on a goldmine of untapped statistical information.   

Let us explain.  Producing the Student Improvement Study was only a small part of our business.  The main part of our business was doing a variety of different types of assessment reports for more than 100 California School Districts.  For almost all of these districts, we would receive their SAT9, CTBS, and SABE results on disc or CD.  We would decode these results for each student and provide specialized reports for the school.  As a matter of fact, we still do this type of work as you may see by looking at the other offerings on this website. 

By 1999, we had an archive of more than 2 million student record lines in digital format.   Furthermore, we had year-after-year longitudinal and cohort data right under our noses.  Before we tell you what we did with this, we would like to explain the significance of what we had. 

We had, beginning in 1997, what the California Department of Education is only striving to have in 2004.  You may be familiar with the CSIS project.  The part of the project most familiar to administrators is that CDE is requiring all school districts to have unique student identifiers for every student in the state by 2005.  The lesser-know part of the project is to form a longitudinal database with all student assessment results.  This had also been scheduled for 2005, but due to budget cuts and other considerations, it appears that this part of the project is now on the back burner. 

At EdTech Associates, however, we have had our longitudinal database since 1995, and it has now grown to more than 6 million student record lines for all types of districts from all parts of the state. The student record lines provide a highly statistically significant sampling of all student records, with a predicted correlation to the whole state student population being r =.82 .

In 1997, therefore, we began testing our studies against the longitudinal database.  We used intense back-testing, back-to-forward period-specific testing, carefully-limited optimization, computerized artificial intelligence, and strict statistical accountability to develop the second generation of our study by 1998.  The new study almost doubled the expected gain in students above the 50^th percentile for schools using it.  Again, we were pleased. 

The year 1999 was a watershed year.  California announced the beginning of the API. High-Stakes testing had begun.  We decided to take another look at our study.

When we took a critical look at our study in 1999, we reached certain conclusions:

The solution to item #4 above was revolutionary unto itself.  By benchmarking to the API growth target rather than to improvement for each student, we determined that the study could work with a relatively small number of students and provide school goal achievement.  This would truly shrink the problem for schools.   Again, our longitudinal database allowed us to devise the proper formula for the number of students to provide for each school.  After seeing the minimum and maximum numbers of students required at our population of schools, we followed statistical rules to give all schools a number of students equal to 2 standard deviations above the mean number of students required.  Thus, approximately 96% of schools will have adequate coverage.  In fact, the number of students for remediation equals about 10% - 20% of the student body.  With this relatively small number, a school may now reach its goal. 

Problem #5 above presented a different kind of problem.  Fortunately, our longitudinal database, our greatest asset, would allow us to tackle the problem.  But the other part of the solution would be scary for us.  EdTech Associates was comfortable with engineers and computer scientists, but to solve this problem, we were going to have to work with educators and psychologists.

In a nutshell here was the problem.  The majority of students are Student A types.  For Student A, our ULTRASCORE Intelligent Remediation Lists would find his weak points that could contribute to overall school score improvement and relate them to the identical Content Standards needed for improvement.  It is logical --- and it worked as expected.  Surprisingly, however, a significant minority of students did not react to the study as Student A types.  We call them collectively Student B types, although it will soon be shown that there are hundreds of Student B types.   

To help Student B , we had to set aside our computer models and speak with psychologists and school administrators.  In a nutshell, the problem we had to solve was this.  Student B tests weak in Content Standard Y.  But if you remediate Student B in Content Standard Y, it doesn’t seem to help.  By backtesting our longitudinal database, we learned that if you want to help Student B understand Content Standard Y, you have to remediate Student B in Content Standard X !!! With the help of the consultants, we determined that selecting the correct Standard X to help with the weak Standard Y was dependent upon understanding the student’s entire personal situation, a situation that consists of:

We therefore decided to classify all students as possessing a certain learning profile which would be a matrix of the above four characteristic sets.  We matrixed 5 intelligences with 6 personal environments with 3 learning environments with 8 scoring strand profiles to hypothecate 720 different remediation profiles.  We then designed an artificial intelligence system of assigning remediation Content Standards on the basis of a combination of the student’s learning profile and the student’s known weak Content Standards.  We put the new concept through extensive back-testing and found a marked improvement in the scoring of Student B types which carried over into statistically significant real-time improvement in overall and subgroup school score. 

This concept was fully implemented in our studies beginning in 2000 and continues to this day. 

In 2004-2005, we re-evaluated UltraScore ™ in light of changes to the API and AYP scoring formulas due to the inclusion of CST and CAHSEE, Social Science, and Science.  We did extensive statistical testing and determined that we could still confine the remediation lists to ELA and Math, but that we would have to provide remediation lists for CAHSEE in some cases.