Tk20 - Assessment Solutions at Work™

Find us on LinkedIn
Find us on Google+
Let's Get Started. Call Us Today.

NCATE/CAEP Accreditation & You

THE CAEP ANNUAL REPORT: DISCOVERING DATA 'HAVES' AND 'HAVE NOTS'


The educator preparation profession is in a data gathering mode as never before. In an era of belt tightening, states, districts, prospective students, parents, the public, the accrediting body, and the federal government all want to know: how effective are teacher preparation programs? Is tax money being spent wisely at public institutions? How many teacher preparation program graduates succeed in gaining teaching positions? What percent decide teaching is not for them and move on to other occupations? How effective are new teachers in the classrooms? Is the state getting a good return on its investment in allocating seats to various programs?

To help answer these questions, CAEP's new Standard 4 requires educator preparation providers (EPPs) to collect data on program impact and outcome data, and eventually, to include these measures in their CAEP Annual Report as follows:

Measures of Program Impact:

  • Impact on P-12 learning and development 
  • Indicators of teaching effectiveness 
  • Results of employer surveys, including retention and employment milestones
  • Results of completer surveys 

Measures of Program Outcome and Consumer Information:

  • Graduation rates
  • Ability of completers to meet licensing (certification) and any additional state requirements (e.g., through acceptable scores and pass rates on state licensure exams)
  • Ability of completers to be hired in education positions for which they were prepared
  • Student loan default rates and other consumer information

These data will shed light in new ways on EPP programs. This new requirement is a real change for educator preparation providers. Annual reports to date have asked for information on a material change in programs, such as a significant dip in budget or a change in the financial status of the institution; many have not asked for detailed data on program impact and outcomes, in part because the institutions simply have not collected or been able to collect that data.

By instituting the change in requirements in Standard 4 and also for Annual Reporting, CAEP is pushing the envelope and stimulating change in data collection targets and strategies.

CAEP understands that EPPs do not have data on some of the above items, and if they do have data, in some cases, the n's are too small to be valid. For example, the EPP may have data on job placement for some but not a meaningful percentage of its graduates.

However, the status quo is no longer satisfactory. EPPs will need to employ new methods to gain access to the data they do not now have, and to increase the response rates for measures for which they have partial data.

Dialog Across Professions Can Shed Light on Successful Data Collection Practices
Colleges of education can take a page from other professional schools on campus to help develop tools and procedures to increase response rates on measures where data is spotty. For example, professional business schools and law schools have long gathered job placement data on their graduates. The American Bar Association provides individual summary job placement reports on its graduates. Simply click on the name of the institution from which you wish data (the complete list of institutions is supplied) and a complete report, including employment status and employment type is available. Those measures are priority consumer look-ups when individuals consider which business or law schools to attend. Having graduates provide working email addresses and coordinating with alumni offices are just two of the strategies these schools use. Cross-professional school dialog could foster innovation and increased success in collecting data on some of the CAEP-required measures.

CAEP Survey: Determining the Haves and Have Nots
For the 2014 year, CAEP is only collecting data on program completers, licensure data (pass rates, etc.), and employment, which EPPs provide for Title II and/or the AACTE database PEDS.

For the rest of the measures for Standard 4, and for Annual Reporting, instead of attempting to collect partial or non-existent data, CAEP is conducting a survey of all providers in its system to determine which of the remaining measures EPPs have and which they don't have. That data collection, occurring now, will inform the 2015 Annual Report and subsequent reports.

More importantly, it will inform CAEP. In the areas where information is available, CAEP will collect that data in future Annual Reports. Where information is not available, CAEP will collaborate with states and EPPs to move toward more consistent and complete reporting. The results will also become part of CAEP’s Annual Report, a recurring report to the public.

Accreditation critics have faulted the accrediting body in the past for not collecting and publishing enough quantitative data on outcome measures. CAEP is responding to its critics, and is also in a better position to require EPPs to collect more quantitative data now than in the past due to recent advances in data collection technology.

An Institution Moves Forward with Support of the State
To gain better information for some of the new data requirements, it makes sense for EPPs to coordinate closely with their states. EPPs no doubt are having conversations with their state departments of teacher education now about the new CAEP expectations. CAEP also has a yearly clinic for states where they can share innovative ideas and best practices to advance the profession.

Tk20, Inc. recently talked with leaders at Portland State University in Oregon about the steps it is taking to improve its data collection on the measures CAEP will expect in the coming years. Many institutions attempt to gain employer survey feedback and placement data, two of the measures CAEP will require. Most are not yet where they want to be in terms of collecting data on these measures, and this is engendering many conversations with licensing boards and state departments of education—in other words, it is stimulating changes to close the gaps. PSU exemplifies these changes occurring now.

Employer and Alumni Surveys
Randy Hitz, Dean of the Graduate School of Education at Portland State University and former chair of the American Association of Colleges for Teacher Education, recently noted that the Oregon Association for Colleges of Teacher Education has begun a project in which EPPs send a joint survey to alumni and employers. "We believe graduate follow-up and employer survey data will be vastly improved as Oregon implements a statewide survey for all teacher ed programs," says Hitz. "We are hopeful that this will lead to better response rates. With this information, we will also be able to benchmark our campus data to statewide data."

Placement Data
Hitz notes that "we do not have good placement data but we are working with our licensure board to obtain the data. The big hole that needs filling here is data on where our graduates work within Oregon. We are hoping that our collaboration with the state will eventually give us access to the data we need."

Dave Bullock, Manager of Technology for Portland State University's GSE, continues on the current challenge: "We can track all of our newly licensed graduates and determine the school districts where they are located. However, we cannot discriminate administrators from teachers from counselors. We also cannot determine the level they teach. We're working on it with the Oregon licensing board now."

The Tk20 System: A Backbone of Portland State University's Reporting
Technology manager Bullock reports that Tk20 works closely with PSU's Graduate School of Education to help the GSE collect the data needed to produce understandable and informative accreditation reports. "We needed a comprehensive system to monitor our students from the moment candidates apply to the program through alumni status. We selected Tk20 because of its ability to customize the technology to our needs and its stellar customer service."

"We are able to collect demographic information that is needed for reports from all applicants (even those who don't end up submitting their application). Data from admissions, the candidates themselves, their test scores, completion rates, aggregate scores on key assessments (including field experiences)—all of this information is stored in the Tk20 system. We are fortunate to have a data specialist who works closely with the Tk20 Data Team. He tailors reports that meet Annual Reporting requirements, and better meet the requests that we get from programs as they review data and make decisions about program improvement," says Bullock.

Here's a snapshot of where Portland State University's Graduate School of Education is with regard to CAEP's Annual Reporting measures and Standard 4 reporting. PSU is a CAEP pilot institution with a visit in 2015. PSU provides an example of where many institutions are with regard to Standard 4 and Annual Reporting measures: in process.

Measures of Program Impact:

  • Impact on P-12 learning and development. Hitz reports, as noted above, that "PSU cannot track our graduates very well let alone link them to student achievement scores. However, the GSE is exploring a better approach with three school districts, which is to tie school district evaluations of teachers to their graduates."
  • Indicators of teaching effectiveness. PSU's GSE has data from student work samples over several years. In the next two years, the GSE will implement edTPATM; this will provide a valid and reliable data point on candidate knowledge, skills, and impact on P-12 student learning. 
  • Results of employer surveys, including retention and employment milestones (see above discussion). "We can now go back seven years with the state data we have, but we do not regard it as especially reliable," notes Hitz. 
  • Results of completer surveys (see previous discussion).

Measures of Program Outcome and Consumer Information:

  • Graduation rates. PSU only prepares teachers at the graduate level; graduation rates are well over 90 percent.
  • Ability of completers to meet licensing (certification) and any additional state requirements (e.g., through acceptable scores and pass rates on state licensure exams). Students cannot complete GSE programs unless they successfully complete the exams.   
  • Ability of completers to be hired in education positions for which they were prepared. PSU has data based on the surveys it has conducted. In the near future, the GSE hopes to be able to use licensing board data to better track all of the graduates who remain in the state. 
  • Student loan default rates and other consumer information. PSU's Financial Aid Office will provide these data.

In Summary...
The CAEP requirements are shedding light on data gaps that have existed for decades. As survey results continue to reveal those gaps, all stakeholders will have an incentive to work together to develop strong systems for collection of important program impact and outcome measures. This will improve the teaching profession and enhance its stature. The profession is on the move.

Read the President's Previous Message about new CAEP standards >