Project Description:
The evaluators will collaborate closely with project administrators and staff to carry-out the data collection. To support accurate and thorough data collection, the evaluators will do the following:
Develop data tracking sheets;
Analyze tracking sheets;
Develop end of event evaluations and/or surveys for feedback for the project activities;
Analyze website analytics;
Analyze event evaluations and feedback surveys; and
Monthly virtual check-ins with project staff to discuss evaluation components.
The evaluators will analyze all data and prepare narrative to be included in the Annual Performance Report required by the Department of Education Office of Special Education Programs.
Description of the evaluation [excerpt from the proposal]
The evaluation will include formative (process) and summative (outcome) evaluation components. Formative evaluation components will be used to examine the implementation and process of the Project in meeting project milestones. Formative evaluation will allow the Project to make changes to the program operation and activities in response to participant feedback on program activities. Summative evaluation components will be used to examine whether the program has achieved all of its intended outcomes, and will examine the impact of the project on:
Increased awareness, knowledge and support through collaboration between organizations/agencies;
Increased knowledge and skill of parents/families and service providers;
Increased engagement with families, especially those from culturally and linguistically diverse groups;
Increased knowledge and evidence of successful secondary transition planning with families and school teams; and
Improved outcomes for students with deafblindness.
One of the first evaluation activities will be to collect baseline data in order to establish targets and refine the performance measures. For each objective, we have identified specific outcomes (including Government Performance and Results Act (GPRA) outcomes) and performance measures (see Appendix A, Table 4) that will provide quantitative data. These results will be reported to OSEP annually through the annual performance report and the Project final report.
Evaluation tools: The evaluation tools will include both quantitative and qualitative measurement tools. Determining progress and outcomes, aligned to the project objectives, requires quantitative approaches describing levels of participation, satisfaction, and implementation progress. Quantitative measures will include surveys, attendance records, aggregate student achievement and participation data, and rating rubrics indicating levels of quality and utility of project materials. Qualitative data will provide deep and rich description from the perspectives of community and state partners, project staff, parents/families, and educators. Qualitative measures will include open-ended questions within surveys, focus groups, interviews, and document/artifact review. To align the project performance measures, project processes and milestones, and GPRA measures, the evaluation tools will be designed to address the following:
Knowledge gained;
Implementation supports and fidelity of implementation;
Quality, usefulness, and relevance of the training and materials provided;
How well the project activities are marketed and resonate with a culturally diverse population; and
Numbers of students and participants benefiting from the project.
This list will be expanded into an overall evaluation plan to include fully developed project measures, data system for organizing data, and timeline for data collection and analysis.
Data collection: Arizona Deafblind Project will contract with a third-party that will create a data organization and collection system, develop evaluation tools, conduct data analysis, and contribute to reporting. Project staff will be instrumental in recruiting participants for the evaluation, distributing data collection links (i.e. surveys), and entering data into the project developed data organization system. Surveys and data entry will be completed online using Qualtrics and links provided by the evaluator. The evaluator will provide regular updates on the counts of completed survey and reminders necessary to ensure attainment of a comprehensive sample. Overall, steps will be taken to ensure reliability, accuracy, confidentiality, and comprehensiveness of data collection. Qualitative data will be gathered through open-ended survey questions, interviews, and focus groups. The evaluator will develop the interview and focus group protocols and collaborate with project staff to determine the most efficient approach for conducting data collection while minimizing bias.
Analysis: Analysis of qualitative data from surveys will involve coding for themes, and identifying themes as reported across multiple respondents. Analysis of quantitative data will examine trends of increase and decrease, participant response to project activities, and output counts of participation. This approach will provide an overall view of the effects of the Project as well as project-specific progression and outcomes. The data will be used to inform the scope and design of project activities.
Use of Evaluation Results to Inform Implementation
Evaluators will develop a full evaluation plan with timelines for development of project-specific evaluation tools, data collection and analysis schedule, and roles in data collection within three months of the new funding cycle. Project staff and evaluators will establish a timeline for reviewing evaluation results so that the data can inform implementation of project services and supports, communication, and outreach activities. At a minimum, review of available data will occur quarterly per year. This is critical in the early phases of the project year 1 when the data-based system is developed and implementation begins. Results will be reported annually as required to OSEP.
Unserved or Under-served Populations:
Disadvantaged Circumstances, Limited English, Specific Groups
Primary Target Audience Geographic Descriptor:
Mulit-County, State, National
