Centralizing EdTech Evaluation and Student Data Privacy Compliance

Oak Grove School District, located in the southeastern corner of San Jose, California, serves TK-8 students with a mission “To ensure every child’s potential is achieved.” The district’s Education Technology Team consists of three edtech coaches who work directly with teachers and site administrators to support classroom technology use.

Inefficient Student Data Privacy Management

At the outset, Oak Grove School District (Oak Grove) was looking for a more efficient way to meet its student data privacy requirements, including those required under California’s Student Online Personal Information Protection Act (SOPIPA). The EdTech Team had been using spreadsheets to keep track of approved edtech tools, but found the process cumbersome to manage and, ultimately, not user-friendly for teachers. The team needed a way to provide more information on individual edtech tools, make it easier for teachers to search for appropriate tools and communicate where tools were in the approval process. Additionally, they wanted to increase their efficiency with managing and reviewing teacher requests for new tools.

Lack of Quantitative Data

Oak Grove educators are invested in making the best decisions for students, but the EdTech Team recognized the challenges around identifying hard data on actual product usage and effectiveness.

Often, the only way decision makers could gather this information was using surveys, which were time-intensive and lacked objectivity. Since the available budget is finite, the EdTech Team wanted to provide more context so decision makers were not relying solely on anecdotal data. The team aimed to find a way to balance subjective, qualitative data with more objective, quantitative data, and provide it to local teams and district leaders before budget meetings.

The first challenge was addressed by setting up a library of district-approved tools using LearnPlatform. The EdTech Team benefited from LearnPlatform’s database of over 9,000 edtech tools and the ability to define the process for teachers to request new edtech tools. The team set up a library of edtech tools for the entire Oak Grove community, including teachers, students and families.

With LearnPlatform, teachers can now easily find tools that fit desired content areas and grades, access more information about the tools and even browse through resources to support adoption. Tools not represented in the library can be requested online by teachers, which kicks off the team’s review process. Teachers have visibility into where each request is in the process at all times, streamlining communication and offering a new level of transparency. Likewise, sharing relevant information with student families is simple, with a public-facing version embedded on the district’s website

The team also looked to LearnPlatform to address its need for evidence, having first experienced LearnPlatform through its work with the Silicon Valley Education Foundation evaluating math edtech products. Using IMPACT™, LearnPlatform’s rapid-cycle evaluation engine, Oak Grove wanted to generate deeper evidence of product usage across school sites. The team aimed to better understand the impact of tools on student outcomes, and wanted to provide decision makers at both the district and school levels with objective data to drive financial and instructional decisions. The EdTech Team’s key goal was to move beyond an aggregated usage report to answer the questions such as:

The team picked two products for its initial IMPACT work: one that is paid for and provided by the district, and another that is purchased using local budget dollars but that is widely used across all school sites and supported by the district.

For the district-provided tool, the team identified a product intended to be used throughout the year for supplemental learning, as well as for formative assessments, three times a year. Oak Grove leaders wanted to better understand how the program worked last year for students to guide plans for the future. The team also wanted to build its competence using IMPACT to evaluate program effectiveness. It opted to run both a Fidelity Analysis, a type of Usage Analysis that includes an intended utilization goal, and an Outcomes Analysis, which is the analysis that evaluates effectiveness. Three types of usage metrics were considered: minutes on system, lessons passed and lessons completed.

Ultimately, when considering the various reports, the team noted that the time spent on the product was roughly equivalent to the time needed for the assessments alone. Since the assessments were mandatory, they inferred that usage outside of the assessment periods was lower than desired. This evidence will be part of planning conversations and budget meetings, and will inform discussions about whether the product still fulfills both its intended uses – formative assessment and supplemental learning.

The second evaluated product had been in use for decades, and was ingrained in district culture. The team felt it was time to pause and review product usage, both generally at each school and across grade levels and classrooms. From there, they considered ROI for each school out of the dollars spent at each school site. Because individual schools all paid separately for the program, the district wanted to explore potential savings opportunities for future collaborative purchasing, something school leaders had been resistant to in the past. To accomplish this, the team ran separate rapid-cycle evaluations for each school site and will use the detailed reports to inform conversations with site administrators, help guide professional development plans and explore opportunities for collaborative purchasing in the future.

The EdTech Team is excited about using rapid-cycle evaluations to provide value to the schools and classrooms they support, both by tying evidence to the professional development they provide and bringing quantitative data to business and instructional decisions. In particular, they see potential for evaluating edtech tools bought during the pandemic to fulfill an immediate need. IMPACT gives them a way to assess whether the purchase justifies the cost and ensure alignment between what people think is happening with a product in terms of effectiveness, and what is actually happening.

While there are unique funds available this year, the team knows that there will always be tough decisions to make. It looks forward to providing critical context to conversations about which programs to keep or cut when the time comes. As Neff says, “It’s not really tangible until you have to make a decision.”