A Date with Data - SoundSage

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!You can contact us via the Podcast page on the IDC website at https://ideadata.org/.### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.34 >> Hey. It's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.50 >> Hello. Welcome to "A Date with Data." On this episode, which is part of a series that we are doing on IDC Data Quality Peer Groups, we are focusing on the SPP/APR Data Quality Peer Group. This peer group is part of a number of other groups that are facilitated by IDCTA providers to bring state-role groups together to discuss and collaborate around data-quality issues that are of greatest importance in the States. In this episode, I am joined by the group's facilitators, Nancy Johnson and Chris Thacker. Thank you both so much for being here.

00:01:00.93 >> My pleasure.

00:01:01.90 >> Our pleasure.

00:01:03.67 >> Great, so to to get started, for those who might not be familiar with Data Quality Peer Groups, and in particular, the SPP/APR group, can you tell me about this group? Who tends to participate in it? What is the structure or the format like, and what are some of the topics that you tend to cover?

00:01:24.06 >> Well, as the name implies, we are the state performance plan, annual performance report, Data Quality Peer Group, so naturally our topics of conversation are around the SPP/APR and the various indicators that are addressed within that document. There is also discussions around the SSIP or the SiMR that are part of Indicator 17. We talk about many different things. The group consists largely of Part B data managers, the SPP/APR lead, indicator leads, the SSIP lead or SiMR leader, depending on what the particular topic is on a call for that particular month. We meet on a monthly basis every ...

00:02:06.71 >> Second Wednesday of the month.

00:02:07.78 >> ... second Wednesday of each month at 3 p.m. Eastern Time through a Zoom link, and we try to let them know ahead of time what that month's topic of conversation is going to be. And really it's a lot of dialogue between and among the the state participants who attend. As the name implies, we're trying to share that information among one another, talk about things that have been of concern that particular time of year. For example, we just had a conversation around the SPP/APR feedback that's occurring, so depending on the time of year as to what the topics might address.

00:02:43.16 >> And what are some of the common topics that have been bubbling up recently that you're hearing from states on these calls?

00:02:50.80 >> Currently, some of the common themes that we're hearing, questions about stakeholder engagement, and in fact, that is our topic for tomorrow's meeting. And we're really going to focus on including efforts to build the capacity of a diverse group of parents to support implementation activities designed to improve outcomes as part of that stakeholder engagement. We also get a lot of questions about representativeness, non-response bias and sampling plans as they relate to Indicators 8 and 14. And of course, this year we're getting questions and issues around collecting and documenting data for the new Indicator 18, which states have to report on in the next SPP/APR in February of 2025 along with that expanded general supervision section in the introduction that includes eight new elements that states are expected to address and then the continuing concerns questions about methodology that's reasonably designed for Indicators 4A and 4B.

00:03:57.92 >> Mm-hmm.

00:03:58.38 >> Those are the things that come to the top of my mind. Chris may have some other things to add.

00:04:02.97 >> I think you got the major highlights there, the topics that come up. There might be some specific things that will happen that are unique to a state or circumstances that exist there or circumstances that a state may think are unique in what to bring to the group to see if other states are experiencing, and that could just be a wide variety of different topics, and that's largely dependent upon what we're talking about. If we happen to be addressing Indicator 4, then we might have something around discipline and how data around that are collected or things of that nature, but it's kind of topic-driven mostly.

00:04:36.64 >> Yeah. One of the nice things about these groups I have found is that it's a great platform for states to come and just be able to ask other states, "We're experiencing X, Y or Z. Is this something you've also dealt with, and if so, how have you addressed it?" or just to kind of feel like they're not alone in some of the challenges and things that are going on.

00:04:58.20 >> That definitely seems to be the way our groups flow. We try to keep it as informal as possible so that we can encourage those open conversations and dialogues among the states.

00:05:08.05 >> Mm-hmm.

00:05:08.36 >> And we get a really good participation, it seems, each time we have a call. Generally, we're talking about anywhere from 20 to two dozen or so, maybe as many as 30, 35 sometimes, participants on board to have these conversations. We try to start it out with providing a little information or a little content, if you will, about whatever the topic of that month's call is going to be. And of course, they know ahead of time, generally speaking, what the conversation or the topic is going to be as well.

00:05:35.96 >> Mm-hmm.

00:05:36.28 >> So that has them bringing in their thoughts and ideas to the meeting so they're not caught by surprise, and it turns out pretty well.

00:05:44.27 >> Well, I was just going to say, I agree that one of the best things about these meetings is states' opportunities to talk with each other and get ideas about issues that they're having related to the topic or just questions that they have and talking back and forth. It always sparks something that states can take back and use.

00:06:04.19 >> And one of the conversations that I've found or topics that have come up a lot is, "What did OSEP say to you about this particular situation, and have you gotten feedback from OSEP on doing it this way or doing it that way?" So those types of conversations really seem to be of the most interest to the group.

00:06:21.49 >> Yeah. You mean states will say, "I had a conversation with OSEP and discussed this particular topic, and this is what they said."

00:06:27.62 >> Yes.

00:06:27.74 >> Yeah, I think definitely that's a lot of states ...

00:06:30.08 >> Yes.

00:06:30.17 >> And I think about that Indicator 4 and some issues that were going around with how you're supposed to collect data and interpret the data for Indicator 4 was a a good conversation that wanted to know what OSEP's perspective was on a lot of those circumstances.

00:06:45.12 >> That's definitely one of the benefits of those calls for states. What are some tips or recommendations that you can share related to the SPP/APR that states maybe that aren't always part of these calls might benefit from from knowing about?

00:07:00.89 >> Well, one of the things that comes to mind, and Chris is certainly the expert in this area, but are getting states to use the IDC's protocols for documenting data processes for the 618 data collections in the SPP/APR indicators and contacting their state liaison to get assistance with that from TA providers facilitating that process. Chris certainly does a lot more of that than I do, but it really does turn out well when states document their processes.

00:07:33.42 >> I think that's a good point, Nancy, in bringing up the the documentation of state protocols. That is one of the big key PA things that IDC does with states. We do a lot of different types of PA, but that one seems to be a hit just about everywhere we go because it brings different people from within the state together at the same table to talk about conversations, about specific topics, whether we're looking at an indicator or specific types of data, whether we're talking child count or discipline or exiting or what have you. Having those people at the table really opens up those dialogues and those conversations and helps people within the SEA to understand what other staff are doing so they have a better sense of the overall picture and not just maybe their particular corner of the world, if you will.

00:08:19.95 >> Just about the data processes, one thing is, I know a lot of states are are interested in them, especially as DMS is is coming up because part of what happens during those DMS visits is really kind of sharing a lot of detail, right, with OSEP around processes. And if you have it all ready to go and documented ahead of time, then you know you have a lot of that already there at your fingertips. And also, given what Chris was saying, it's great to get down on on paper, electronically, all of those detailed steps. But a big benefit of doing the documentation, to Chris' point, is that it brings together a number of folks who maybe don't often come together and have these discussions. So it has sort of that other added benefit of really having that dedicated time to sit down and hear what everybody is doing and really be able to understand at a higher level, from start to finish, everyone that's involved from the collection, analysis, reporting, every step along the way.

00:09:25.70 >> And I'm glad that you brought up the DMS 2.0 and the impact that that might have with regards to documenting processes. You may not be documenting your processes for the purposes of doing DMS 2.0. Ideally you're doing it to enhance the quality of the data that you're capturing or to have consistency in how you might fill out your SPP/APR on particular indicators or to break in new employees to give them some information.

00:09:51.13 >> Mm-hmm.

00:09:51.70 >> But if you've got processes documented, it's going to be a very big help when you go through the DMS 2.0 interviews and conversations that you would have. But those that you have a place to reference to, somewhere to go back to so you don't get lost in those types of conversations, if you're familiar with your protocols, it's going to become more second-nature to you in answering those questions. So I think it's very, very helpful for that as a side benefit. But to me, the primary reason for doing it is to enhance your own activities and your own processes to have that consistency.

00:10:27.42 >> Mm-hmm.

00:10:28.10 >> One other benefit with the data processes, it's a way for more than just one staff member to understand what those processes are because it is documented on paper, and so staff have a better understanding of the processes that go forward rather than one person just having all that information in their head or to themselves so that it really becomes a team effort, as we've discussed earlier.

00:10:58.05 >> Another benefit that hits me, and it's not something that's discussed a great deal, at least in my mind, state directors come and go from states on just out of normal transitioning. And if they come into this role and can see all the processes that are going on within the office, it can give them a better appreciation and understanding of what roles and responsibilities are of individual staff as well as the office at large. It also can help to develop an appreciation for data managers. Therefore, the state director have an appreciation for that data manager and to know what it is that they're providing and kind of gives them a go-to person, if you will, for different topics.

00:11:39.86 >> Those are all very true. Any other tips or recommendations?

00:11:45.49 >> Something else that comes to mind for me is developing an annual plan for when you're sharing your data, when you collect and share your data for use with specific indicators for the SPP/APR so that you should, in my opinion, be scheduling regular meetings between your SPP/APR coordinator, your data manager, your indicator leads and any data analysis tied to certain indicators as appropriate so that throughout the year, working on the development of your SPP/APR rather than waiting until the last couple of months of the year soon before it's due. Because some of this data that feeds some of the indicators you have early in the year, and then some of it you get later in the year during the summer. So it gives you a time, if you develop that annual plan, to figure out how you're going to plan out your work and then meet regularly to make sure everybody is on the same page about what the data is saying, what you're going to be using related to the SPP/APR. So I just think that annual plan helps you with organizing all of that information.

00:12:58.25 >> I would also like to add that being part of the community through the Data Quality Peer Groups gives you the ability to network with your peers, and oftentimes the peers that are doing the same type of work you're doing in a different state. So that can broaden your knowledge base of places to go. Obviously being that Nancy and I both work with the IDC, the Data Center, we are state liaisons for particular states. You can come to us, your state liaison, ask questions or concerns. It might not be that particular liaison has the answer to your question, but we can work with our colleagues at IDC to find that answer, and sometimes we might even reach out to other TA centers funded by OSEP to be able to get answers for you. So don't be bashful about asking your state liaison for assistance around any of this. Don't be bashful when you're attending your Data Quality Peer Groups to ask questions. Yep. That's how we learn, is sharing the information and experiences that we each have because they're all going to be different, but they might be close enough to give you an idea that you might not have thought of on your own, and seeing how another state is doing something could give you an idea to go back and and do something new and different back home.

00:14:08.51 >> On all these groups, we encourage any and all questions because it might be something, like Chris said, you haven't thought of before that kind of jogs an idea in your head, and we try to make these very ...

00:14:20.34 >> And I would add there, too, we do not record these.

00:14:24.55 >> Mm-hmm.

00:14:24.95 >> We do not have people from our staff from OSEP attending. So it's kind of an anonymous group, if you will, anonymous in that there's no one there who is a monitor over your programs that you can come in, and from the information you shared, be able to do something that ... in a monitoring capacity because that's not our role as a TA center. Our key role as TA center staff is to provide you that technical assistance that you need to be able to be in compliance and operate your programs appropriately and to improve results for children with disabilities. It's not a gotcha place. It's a safe environment, and it is your colleagues for the most part that you're talking with. We just kind of provide the space, the opportunity, for it to get you all together and have those conversations.

00:15:10.66 >> I'd like to put in a plug for one other IDC tool that I found helpful when I was an SPP/APR coordinator in a state, and that's IDC's Data Meeting Toolkit. You have, as an SPP/APR coordinator, the data manager. You have a lot of meetings around your data and around developing your SPP and in terms of analyzing your data for that, those kinds of thing, particularly when you have meetings with more external groups, stakeholder groups. That meeting toolkit was very helpful to me, and I hope it's helpful to states in helping plan for those meetings. The tools within the toolkit were just very beneficial, so I wanted to put in a quick plug about that because this is ... It's about the SPP/APR, but it's all about the data that you use and how you analyze your data to make decisions for the SPP/APR.

00:16:13.90 >> Thank you both so much for sharing all this information and letting us know what this group is like, what are you discussing on these groups, and your tips and recommendations.

00:16:24.13 >> Thank you for having us.

00:16:26.77 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:07.34 >> Hey, it's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.65 >> Hello, welcome to "A Date with Data." We are kicking off a special series of episodes that are focusing on IDC's data quality peer groups. These groups are facilitated by IDC TA providers to bring state role groups to discuss and collaborate on the data-quality issues that are of greatest importance in states. For this episode, I am joined by the facilitators of the Data Manager Data Quality Peer Group, Kelley Blass and Kristen DeSalvatore. Thank you so much for being here and welcome.

00:00:56.74 >> Thanks, Amy. Thanks for having us.

00:00:59.78 >> Yes, thanks, Amy. It's a pleasure to be here.

00:01:03.13 >> So to start things off for those who might not be familiar with these peer groups, can you tell us what these peer groups are like? Who tends to participate in this particular group? What is the structure and the format like? And what are some of the topics that you cover?

00:01:19.76 >> The Data Manager Data Quality Peer Group is a space for folks to come together to discuss and learn all things IDEA data. So as IDC facilitators, we provide timely information and resources, but there is also sharing by the states and learning from each other.

00:01:41.45 >> I agree with that, Kristen. I feel like though we do discuss topics and come in with data topics each month, the bulk of the learning really comes from the discussion and the information that states are able to bring forward, so we'll set the stage, and then they really take it from there and support each other. So it's also an opportunity for real networking.

00:02:10.15 >> The name probably says it's data managers that are a part of this group.

00:02:15.25 >> Yeah, so all IDEA Part B data managers are welcome.

00:02:18.92 >> Mm-hmm.

00:02:19.56 >> But they can also invite and bring other staff in the SEA that they think will benefit from participation. The meetings are held virtually on Zoom once a month, and as Kelley mentioned, we set the stage, so IDC comes prepared with a structured PowerPoint, but as we also already said, there is ample time and encouragement for interaction and dialogue with us, IDC and with the other states that are on the calls. So IDC is really there to support the states, so discussion and collaboration on all IDEA data subjects are open to discussion whether they are on the agenda or not.

00:03:13.49 >> But we also give states an opportunity to share their resources, so, for example, if we're talking about one of the 618 data collections, and they have a state-developed resource, we love to give them an opportunity to share their resource, which then could also turn into another opportunity for states to network together and collaborate on resources. And states that use IDC-developed resources are also often invited to share how they use it, demonstrate the use of it and share how they may use it either for pre-edit checks, so if they're using an edit check tool, they may share how they use that for their edit checks and for their data quality, or they may share how they use it for their public reporting. But we really like for states to come in and demonstrate the use of their own tools as well as IDC-developed tools.

00:04:09.88 >> I know when I was involved a while ago with those groups, that was one of my favorite parts and I think the states' too was getting a chance to see how other states were tackling some of the same challenges, what they were coming up with in terms of solutions and resources. They can kind of piggyback off each other. I know in some states they might even have reached out after they saw a particular resource, reached back out to that state and said, "Hey, can we hear more about this? We'd really love to maybe adapt something like it for our state." One of the best parts of these groups.

00:04:44.44 >> Yeah, that's always ... That's wonderful when states ask for contact information of each other and plan to follow up offline to take advantage of knowledge in one state that could be useful in another state.

00:04:59.92 >> And thinking about that, another really positive aspect of these groups is that it does feel like a safe space for data managers to share that information and talk about any of the concerns that they may be having with their data, whether they're having errors or even just sharing their resources because those calls are not recorded, and it is a smaller group. They may be more apt to share and discuss openly than the might be if we were trying to invite them to present at a nationwide conference or something, so we're finding that, that safety and that small-group feeling tends to open up more discussion.

00:05:41.63 >> Yeah, definitely.

00:05:43.13 >> In terms of some of the topics that we cover ...

00:05:46.02 >> Mm-hmm.

00:05:47.15 >> ... We really do cover IDEA data collection and reporting from A to Z., right, from the collection and requirements of the collections to the actual submission of the data. In particular, reporting requirements, of course, the SPP/APR is always a big one. EDFacts Data Files timelines and processes are some of our frequent fliers, and something that Kelley and I do is, we work really hard to make sure that folks are aware of all of the resources and services that IDC has available to states. We find that sometimes there are states that just aren't really aware of everything that is available, and the resources and services can really be amazingly helpful to states, so we find that it's really important that states realize what is out there and do our best to share things.

00:06:54.32 >> And one of the things that we've noticed from that sharing, and so in our just common template that we use every month, we touch on the data selection calendar that's posted on IDC website, and we also make sure that we bring up the comprehensive list of resources, and we've found that each month, that there are a-ha moments or discussions about some of the resources that are available. So that's been very helpful to just keep that as a forefront, and one of the first things that we discuss is what's upcoming and then touch on those, that comprehensive list.

00:07:34.44 >> So being very ...

00:07:35.01 >> Yeah.

00:07:35.38 >> ... relevant about what's going on at that time of year for the states and making sure that, that gets covered or at least brought up as a topic that ...

00:07:45.55 >> Right.

00:07:45.67 >> ... folks would have questions about or want to get into.

00:07:48.31 >> Absolutely, and really to provide a service to the states to keep them focused and kind of take a tiny bit of the burden off them to keep them on track, it can be very difficult as a Part B data manager to keep all the balls in the air.

00:08:06.32 >> Yeah, and know what's ... everything that's out there, like you said. IDC has a lot of resources. All of the centers do, and it seems like these groups are able to sort of elevate and point out the ones that are most relevant at that particular moment in time, which, like you said, kind of takes away some of the burden of the data manager having to pour through websites and be looking for things themselves.

00:08:30.60 >> Yes, they don't know what they don't know.

00:08:34.28 >> Very true, so you mentioned more generally a number of topics more generally that you cover, but what are some of the common themes that have been emerging in the last couple months of calls that you've held with this group?

00:08:52.39 >> Well, certainly in the last couple of months, SPP/APR-related issues.

00:08:57.30 >> Mm-hmm.

00:08:58.83 >> There has been a lot of discussion and focus on gathering the data, the process that's used to write the APR, if there's any kind of submission issues, responding to OSEP-required actions and then of course clarification, which has just finished. Some other things, frustration and confusion with the differences between significant discrepancy and significant disproportionality.

00:09:32.28 >> Mm-hmm.

00:09:33.56 >> And the reasonableness of indicator for thresholds, those have really been a topic of discussion lately as well as we've really dove pretty deeply into documentation and processes in general.

00:09:54.52 >> Right, and particularly around the documentation and processes, we've had states talk extensively on how helpful it has been for them to access the resources at IDC for their liaison to come out and document their data processes when they're thinking about their DMS 2.0 work, and in addition to what Kristen stated around SPP/APR and Indicator 4, those questions seem to come up pretty much monthly. There's also often discussion around Indicator 18 and general supervision, thinking about the new things that have come out over the past year and how they're going to collect information. Everybody is ready for the Indicator 18 calculator to come out, so they're really looking forward to ii24, where they're going to be able to see some of the new things that are available to support them and their data.

00:11:00.41 >> A lot going on around the SPP/APR and changes that are coming soon. That's a hot topic, I'm sure.

00:11:10.30 >> Yes.

00:11:11.72 >> Well, and then the other thing that we have found to be just a really, really big topic is that a lot of states are in the process of either developing or redesigning their public reporting dashboards, and so we've had states come on and share what they've developed. We've had states come on and share kind of partial development and ask questions about, how should we present a certain indicator? For example, Indicator 7 is a really rough one, and because there there has so many components and categories to it, how should they present it within the BI tool that they're using? And so there's been a lot of collaboration between states talking about those tools and talking about how they want to present their data publicly, so that has been very rich discussions, and we've found, like we said before, a lot of states are finding opportunities to engage with each other after the calls and say, "Oh, I really like what Georgia is doing, so I need to have discussions later on with their data manager, so that's been a very rich theme that's been coming up.

00:12:19.16 >> And using those calls, too, it sounds like for states to kind of get feedback from their peers who are engaged in the same thing in their state, kind of ideas and what we started on the right track. What ideas do you have for how to continue the work? Those sorts of things, that's kind of another side benefit.

00:12:38.34 >> Absolutely, we had a rich discussion at our last meeting about how to make the data more digestible for stakeholders.

00:12:50.20 >> Hmm.

00:12:51.01 >> Right? And it was a really great discussion with different states chiming in on what they do and struggles that they have had with presenting data to stakeholders.

00:13:04.36 >> Mm-hmm, yeah, that's definitely something that comes up quite a bit, especially around SPP/APR time.

00:13:10.64 >> Yep, that is going to be part of our topical discussion at ii24.

00:13:16.35 >> Great.

00:13:16.90 >> That's what I was going to say. Yeah, that was such a rich discussion. That that's what our data managers chose as their topic to continue that discussion.

00:13:27.45 >> Great, looking forward to hearing that. What are some tips or recommendations that maybe you've shared during these groups, resources, things that maybe you all as former data managers have found to be helpful that other states might benefit from knowing about kind of related to the 618 SPP/APR data, the data-quality topics that come up in this group?

00:13:53.83 >> So as Kelley already mentioned, we have had several states share Data Dashboards and visualizations, and that has really been amazing and helpful. It's been really cool for states to see what their colleagues in other states are doing and for states to be able to ask questions and have a discussion around how states are making it happen, as Kelley said. What ... software, who does the work? What is their role? Et cetera.

00:14:27.61 >> Mm-hmm.

00:14:28.33 >> So that has been ... It's not a specific tip or recommendation, but it's been just great knowledge-sharing in general. And then of course tips on APR language and submission are always welcomed and found useful. At our last meeting, we had a discussion, as I mentioned, about making the data more digestible, and one of the states shared information on a product or app that's called the Hemingway Editor, and it helps you tell a story and how you can look at data in real time, edits in real time and tells you what's hard to understand and I believe even what grade level things are written ... that it's a very specific tip ...

00:15:27.38 >> Yep.

00:15:28.04 >> ... [Indistinct] and the result of it, of being in the group. And then I wouldn't say another huge bonus in the, quote, tip category is, as we've already said, learning about resources and services that are available through IDC. There's so many offered that I was not fully aware of when I was a data manager, and I do really regret that I didn't participate in the Data Manager Data Quality Peer Group more often when I was a data manager in a state.

00:16:08.82 >> I totally agree with that, and I feel like the value in being able to align those resources with the specific topic that we're talking about is very helpful to data managers. We also really try to listen for any changes that are coming up with 618 data, 616 data submissions, if there's going to be any changes in the EDFacts file specification. For example, we heard of a change in language in FS009, and so our first meeting after that change, we kind of highlighted it, and then of course the data managers had some questions, so we went back to OSEP and to some of our TA providers at IDC and asked some questions, and so now at our next meeting we'll be able to provide some more clarification. So we really try to listen to what's happening in the moment and real time so that we are addressing questions and concerns that they're having right now.

00:17:10.48 >> And sometimes things come up not expecting. Again, I'm going to reference our last call that we just had last week, and we had a participant from one of our entities who stayed on after everyone else had left to ask a question about EDFacts reporting that turned out it was not IDEA-related.

00:17:35.80 >> Mm-hmm.

00:17:36.44 >> But I was able to point him in the right direction and give him enough information to get going, so it really is ... It's just a great place to come to for support.

00:17:50.55 >> Thank you, Kelley and Kristen, so much for being on. You shared such great information about this wonderful group, really appreciate your time.

00:17:59.11 >> Thank you. We appreciate the opportunity to share.

00:18:02.86 >> Yes, thank you so much. It's been great.

00:18:07.93 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You are listening to A Date with Data with your host, Amy Bitterman.

00:00:07.34 AMY BITTERMAN: Hey, it's Amy and I'm so excited to be hosting A Date with Data. I'll be chatting with state and district special education staff who just like you are dealing with IDEA data every day.

00:00:19.50 >> A Date with Data is brought to you by the IDEA data center.

00:00:24.60 AMY BITTERMAN: Hello. Welcome to A Date with Data. Now that states have used EDPass to submit all of their 618 data files, we want to hear how those submissions went. And in particular, highlight how the data manager and EDFacts coordinator in one state have collaborated to successfully submit the files using the new system. On this episode, I am joined by Dominique Donaldson, who is the Part B Data Manager, and Adam Churney, who is the EDFacts Coordinator, and both are with the Georgia Department of Education. They're going to share about their experiences and provide some strategies that worked for them to partner effectively in this process. Welcome Dominique and Adam. So glad to have you on.

00:01:05.27 DOMINIQUE DONALDSON: Hi Amy. Thank you so much for having us.

00:01:08.88 ADAM CHURNEY: Yes, thank you for having us here.

00:01:11.10 AMY BITTERMAN: So I want to start off, if you each can just very briefly introduce yourself, say a little bit about your role at the Department of Education. Dominique, do you want to go first?

00:01:20.62 DOMINIQUE DONALDSON: Hi, my name is Dominique Donaldson and I am the Part B Data Manager for the Georgia Department of Education. And I've been with the Georgia Department of Education for a little over one and a half years.

00:01:33.88 AMY BITTERMAN: Great, thank you. Adam?

00:01:36.80 ADAM CHURNEY: Hi, I'm Adam Churney. I am the Georgia Department of Education State Data Analysis and Reporting Manager, which includes being the EDFacts Coordinator. I also handle Power BI Reports and many other data-related stuff.

00:01:50.53 AMY BITTERMAN: All right, so going back a few years now, when EDFacts Modernization kind of first came up and knowing that it was happening, why was it important for you all to approach those changes that were coming as a team and how did you create that team?

00:02:08.41 DOMINIQUE DONALDSON: Well, you know, for the longest time we had been hearing about EDFacts Modernization and there was so much talk about what was coming. It was discussed and shared in emails from PSC and IDC, you know, in all of the conferences like the Interactive Institute and OSEP. But there were very few tangible resources about EDFacts Modernization. So, you know, it kind of made us a little nervous. So in preparation for the EDFacts Modernization, the EDFacts coordinators and Part B Data Managers participated in all of the webinars and working groups. And not only was this process changing, but there was an entirely new platform for submitting the data files. So it was very, very clear that collaboration was going to be the key to a successful transition to the EDFacts modernization using the EDPass system. So, you know, we knew that we needed to work together.

00:03:09.40 AMY BITTERMAN: Mm-hmm.

00:03:09.92 DOMINIQUE DONALDSON: And we needed to first identify the necessary people required for the task. So the EDFacts coordinators identified Part B Data Managers as integral partners in the submission of special education data. So the Part B Data Managers were provisioned for access and relevant collections in EDPass. Our EDPass coordinators were crucial in helping the Part B Data Managers acquire the correct provisioning through the ed.gov and EDPass platforms. Not only did they grant us the access, but there was also a strategic walkthrough by our EDPass coordinators of the system to orient us to the inner workings of the system. In that walkthrough, we were able to navigate throughout EDPass and ask questions. Being able to access the system, interact with the system components such as the file download, and to be able to see the options within the platform allowed the Part B Data Managers to be a relevant part of the data submission process. You know, looking at the system together really helped us to visualize the shared goal that we were working to complete together and to ensure that we submit accurate and timely data for students with disabilities.

00:04:39.63 AMY BITTERMAN: Thanks, Dominique. So Adam, tell me how you as the EDFacts coordinator, and Dominique as the Data Manager, how do you share information back and forth?

00:04:50.52 ADAM CHURNEY: Sharing data, there's a lot of places where we actually share data because it really starts when we collect the data from the LEAs. We have several data collections and in each one of them, we've got reports that basically are already out there that we can do our first check on the data to make sure that it's coming in correctly. From that, we, most of our data that we send to EDFacts is actually aggregating from these data collections. So in that process, my team has created a portal within our application that we have all our data collection in that allows the data managers for IDA and all the other ones to basically go in and start seeing these aggregated reports, aggregated accounts, not just for the current year, but also a comparison with previous years to do that first check of data quality.

00:05:35.44 AMY BITTERMAN: Mm-hmm.

00:05:36.00 ADAM CHURNEY: And within that application, we've also embedded our ability to sign off on the data for the data managers. So not only are we getting the data in, but then the data managers are really going through looking at the data and signing off on it.

00:05:49.93 AMY BITTERMAN: Got it. That's great that you have that shared application and can do all of those checks like to your comparisons it sounds like, as well as some of the other validations I'm assuming that get done through EDPass as well. How do you, between, again, data managers and the EDFacts team, how are you collaborating and working together to really make sure that the data that you are submitting are accurate and of high quality?

00:06:15.78 ADAM CHURNEY: So one of the benefits of the new EDPass system is the ability to check your data quickly.

00:06:21.70 AMY BITTERMAN: Mm-hmm.

00:06:22.06 ADAM CHURNEY: So one of the priorities as the EDFacts coordinator is we really want to get the data into the new system as soon as possible. There's that first round of data checks. First is the format correct, which there's format changes every year.

00:06:33.81 AMY BITTERMAN: Mm-hmm.

00:06:34.15 ADAM CHURNEY: And most of those are just a simple fix here or there. Sometimes when there's new data that was collected that we're having to pull in as a new element, we go through and do that first check. But once that first round of uploads happen, me and we also have a programmer [INAUDIBLE] that kind of go through that first errors, data quality edits that we get and we go through and determine quickly whether it is a data issue as in the data that we collected is showing numbers that are out of spec or is it the code that we use to aggregate the data causing that. And last year being the first year EDPass was used for a lot of the files. There was a lot of data quality where it was on the code part where we didn't include zeros. And as a team, we really didn't want to also show this to the IDA managers that go like, "Oh my God, there's so many errors." That, "Okay, just wait. Let us do our first checks." And we did those first checks. We'd identify this data quality check is a code error, or this one is an actual data error. So once we did that first cleaning, we had a lot less error message to then take to the IDA managers to basically look at and say, "Okay, here are the data errors that are actually being caused by our data." And then we do that first round of checks where we'd go, is this a unique case to that system, that LEA, that school, or is this a bigger systemic thing with how we collected the data?

00:07:58.63 And then we would kind of turn through those and make sure that okay, it is a unique case for the school. And from that collaboration, I'm not a subject matter expert when it comes to IDE data particularly, but I know enough that I'm able to sit there, go through, say, okay, let's figure this out, get with the IDA managers and really look at the data, question the data, and then be able to report it. And then one of the nice things in EDPass is if we have a problem where we're looking at a data quality issue and it's like maybe we should have coded it as this instead of that. What's really nice about EDPass is we would take our data file, make a couple of edits to the schools or the LEAs that were throwing these data quality issues, submit it with this altered data to see does this cause other issues or does this clear the air? And being able to do that iteration process, we really were able to ask more specific questions in the data to say what happens if we coded it this way and see, uh, oh, something was triggered over there. And then go like, maybe we shouldn't go that way, or oh no other errors triggered.

00:08:59.99 And then we could come back to our IDA team and say, Hey, if we'd make this change, is this better data quality? And we could kind of iterate. And as you got through the big errors and got to the smaller and smaller errors, you could really start getting into good conversations where we're really talking about how is this data being interpreted, how is it being gathered, is it accurate, is it showing the intention what we're supposed to be showing? And it's really helped a lot with data quality.

00:09:25.36 AMY BITTERMAN: So it sounds like for you all EDPass has been a good thing, it seems like. Has it helped you think with the data quality?

00:09:30.99 ADAM CHURNEY: It has helped a lot.

00:09:31.97 AMY BITTERMAN: Great.

00:09:32.23 ADAM CHURNEY: My first years here I was under the old Eden system and it was great, you know, format errors were checked off early and then, you know, several months later I was like, oh, we've got these data quality issues.

00:09:43.13 AMY BITTERMAN: Yes. There isn't that lag anymore. You can kind of deal with it all at one time and get in, like you said, to sort of even some of those more nuanced, you know, issues to really ensure that the data are absolutely, you know, as clean and high quality as you would hope they would be.

00:10:01.24 ADAM CHURNEY: Yes. Over the past year with all the files that we submitted last year, we had a lot of great conversations with individual program managers where we would sit down after we did our first upload or even after those second uploads and just kind of go like, "Let's talk about the data. Are these numbers correct?" It really got more into the specifics where being on the outside of a lot of the data, I could ask the simple questions that they hadn't thought about it and well, and they would go like, "Maybe we should try this." And then being able to iterate, we could upload a file with a small change, see if that caused any problems or anything else would pop up. And it really did help create good conversations. And even with this charter upload that was just happening being the second year, all the errors that we had from the previous years about the data building and the data code are gone. So now we're really down into these specifics and we've had a lot of great conversations where should we be in some cases with the charter data we have multiple management organizations and just sitting there talking through that with the team asking the PSC if we did this, what would happen in trying that. It really helped us kind of get together and communicate and collaborate. Just really focusing on these small little nuances that we never had time to think about in the past because it was, everything's good check and then three months later uh, oh let's fix the data. Now it's like, let's talk about it and fix it and kind of iterate from there.

00:11:26.43 AMY BITTERMAN: Mm-hmm. And you feel like you're maybe not getting dinged as much either as obviously, I guess, because you're seeing those issues come up before the data get officially submitted, which is probably--

00:11:41.68 ADAM CHURNEY: Exactly.

00:11:42.12 AMY BITTERMAN: Nice to kind of deal with those early on before it gets too far down the road.

00:11:47.30 ADAM CHURNEY: And I think just the whole collaboration process while we're actually able to re-upload files is really helping data quality. It really sits there just iterating and making sure that is this data right. We have the time now to really ask that question and dig into it to make sure that what we are sending to EDPass is the best data that we can give.

00:12:07.15 AMY BITTERMAN: And you've already spoken to this, but I don't know if you or Dominique have anything to add in terms of what have you seen as the impact of this collaboration and working so closely together.

00:12:17.22 DOMINIQUE DONALDSON: Well, you know, Adam really touched on a lot of the positive impact in the collaboration. And as a part B Data Manager, it really has made, you know, my life easier because of the consistent communication between the two departments. You know, if we had not formed that strong collaborative nature then we might not have identified issues in enough time, which could have possibly resulted in submission of inaccurate data or late data submissions. And the collaboration has really made it where we are able to identify any issues and concerns prior to. And it allows us to be able to submit all of the files in a timely manner and have accurate data submissions and like Adam said, improve data quality.

00:13:06.21 AMY BITTERMAN: Yes. Which is the ultimate goal of all this.

00:13:08.50 DOMINIQUE DONALDSON: Right.

00:13:09.57 AMY BITTERMAN: So I'm sure there are states listening who wish they had the kind of collaboration and strong partnership that it seems like you all have in Georgia. I'm sure it wasn't necessarily easy to get there and there was a lot of work involved. Can you share with us what are some steps that maybe others states out there can take to try to build that collaboration that you all have?

00:13:31.25 ADAM CHURNEY: Yes, I think one of the first things that we need to do is really identify the necessary people needed.

00:13:36.27 AMY BITTERMAN: Mm-hmm.

00:13:36.89 ADAM CHURNEY: Of course, there's the EDPass coordinator. And one of the things that I think is really important with the EDPass coordinator is instead of programs sitting there going through individual business rules and the BRSI or BSRI, I always get that acronym [LAUGHTER]. Being able to be the translator for those program managers so they don't have to live in that document at all, since throughout the year I'm living in it every single cycle. Just going through what errors are popping up, let's understand them. So that really helps having that center of focus person. The other interesting thing that Georgia has is we have our data collections which collect the majority of all the data that we do submit to EDPass. And then we have the programmer and myself that will sit there and create the files to then upload. So my team managed the uploads. So really the program managers don't have to get their hands dirty with data too much. They have to understand the data and understand their data, particularly with how it relates to each other. So with my team, we'll build the files, we'll upload it as soon as possible. And then once we have that we can then go to the Part B managers and any other program managers with the data and say like, Hey, we've got the data up there early. This is what we found. These are the data errors that are code-related where we need to sit there and kind of recode it. Don't worry about anything else until we fix that. And then once we fix that saying, okay, Part B managers, here's what's truly errors. Let's talk about this. Let's see, is it just a issue where we've got some systems that have weird data or is it something else?

00:15:06.86 But really once we found that team and figured out who's responsible for the data, but it's all kind of that center spoke around the EDPass coordinator making sure the data's getting uploaded, being built in time, and then having the managers bringing them in at the time where their time is used to the fullest where they can sit there and say, this data is correct. I know you know, X School has X number of students what they know. And relay that back to the thought. So that's really the key people that we kind of brought into this.

00:15:36.56 AMY BITTERMAN: Yes, that makes a lot of sense.

00:15:38.30 DOMINIQUE DONALDSON: And, you know, after identifying those necessary people, which Adam just touched on.

00:15:43.34 AMY BITTERMAN: Mm-hmm.

00:15:43.67 DOMINIQUE DONALDSON: Then we, you know, it's going to be important for teams to set up meetings to discuss roles, access, and goals. IDC has a very useful resource called the EDFacts Modernization Planning Questions to Consider. And that research is going to be helpful for teams when we're looking at understanding who's responsible for what and what processes you need to complete the task and how you will collaborate as a team. You can search for this file right on the IDC website and it'll help you answer some of those questions like when will we submit files? You know, reviewing the EDFacts submission calendar organizer is also so very important. Using backwards planning to make sure your data is reviewed and submitted on time. You can also look at other questions and discuss where data comes from and what are the audit checks for accuracy and verification of data. Also thinking about what access will be given to special education. Special education needs to have access to all of the EDFacts files containing special education data.

00:16:48.71 AMY BITTERMAN: Yes. [LAUGHTER] You would think that would be a given. Right?

00:16:51.34 DOMINIQUE DONALDSON: Right.

00:16:51.70 AMY BITTERMAN: Yes.

00:16:51.98 DOMINIQUE DONALDSON: You know, all of that special education data, it really should be reviewed by the people that you've previously identified in special education as crucial to the project and the data submission process. And so it's going to be important that no special education data is submitted without special education teams getting a chance to look at the data first and verifying the data.

00:17:16.21 AMY BITTERMAN: Mm-hmm.

00:17:16.61 DOMINIQUE DONALDSON: I also want to know when and how you'll review those files. So thinking about processes and then what are the procedures if a concern arises? You'll need to know who you need to alert first when you see something that's of concern. And one of our issues in our state, our first contact is for our infotech architect and we bring concerns to him and we work through the issues together to resolve them. Another thing that you might want to do is also make sure that you're reading the current file specs and developing that common language. That way everybody is understanding the same information. And it's going to be also important to set up regular meetings to discuss upcoming collections to ensure that your data is in alignment with the file specifications. And then you also want to use that data submission organizer to set tentative dates for submissions to make sure that you are timely and accurate.

00:18:20.67 ADAM CHURNEY: Another thing that I think really helped us was attending office hours together or whenever we could.

00:18:26.58 AMY BITTERMAN: Mm-hmm.

00:18:27.24 ADAM CHURNEY: The office hours that PSCS put together are immensely useful. Not just sitting there, you know, asking the questions you have, but listening to other SCAS and the issues they're having with their data. A lot of times I would find myself listening to an SCA that had an issue and they were talking through it, then I would go, "Do we have that same issue?" [LAUGHTER] And then I would go to the program manager saying like, "Hey, this was an issue in this state. Is this something that we have, how are we accounting for this?" And it really helped seeing not just your state, but seeing everybody else's problems. And having the IDA manager there too also helps because they're seeing what the other program managers are having problems with. And I can't tell you how valuable they were because I learned so much about other states. And then from what I learned from them, I learned more about our data because again, now I had more questions I could ask our teams about our data to see how the data was actually being formed. And it really helped.

00:19:23.67 DOMINIQUE DONALDSON: And the last thing, we should always celebrate the wins. It's important to make sure that your team knows how valuable they are and they need to know that the work that they do has a powerful positive impact on the data that we submit. So working together of as a team is so important and making sure that we say thank you. We are very grateful for Adam and his team because [LAUGHTER] without them, we wouldn't be as successful as we are.

00:19:59.38 AMY BITTERMAN: Yes. I've worked with Dawn going back many years, one of the former data managers, and Dominique and I've heard over the years, Adam's name come up many, many times, [LAUGHTER] so.

00:20:09.08 DOMINIQUE DONALDSON: Mm-hmm.

00:20:09.41 AMY BITTERMAN: I know [LAUGHTER] how wonderful that relationship is, And I think it's something that a lot of states will kind of envy and hope some of the information you all shared can kind of help get them more towards that type of relationship too.

00:20:24.75 DOMINIQUE DONALDSON: Yes. I agree.

00:20:26.28 AMY BITTERMAN: So what are your plans for the future? What do you have coming up regarding your working relationship EDPass, EDFacts Modernization, what's going to be coming up next for you?

00:20:38.57 ADAM CHERNEY: So now that we have a year of EDPass behind us.

00:20:41.30 AMY BITTERMAN: Mm-hmm.

00:20:41.57 ADAM CHURNEY: And we've gone through most of those code errors that, you know, consumes some time. The plan this year and going forward is, and you can search for whatever infographic you want, but there's that data information, knowledge, wisdom pyramid, where data's down at the bottom, then it gets a little smaller. Information, knowledge, and wisdom. SCAS are really good about collecting data.

00:21:03.04 AMY BITTERMAN: Yes.

00:21:03.40 ADAM CHURNEY: Some of them are getting better at making it into information where you organize that data into a fashion that makes logical sense. It's the last two tiers, the knowledge, and wisdom where some SCAS have better ability to do that. And one of the things that we're moving forward with is we're creating-- we're using Power BI as our informed data decisions. And we're using Power BI as the tool to express that and communicate that. Not just building the reports. What we had in our Eden portal that we talked about earlier within our application, we're just canned static report, here are the numbers, here are the LEAs, et cetera, which are useful. But when you have 200 or 2,300 schools, 230-something LEAs, it gets a wash in noise. So if there's anything that stands out, you just don't see it in a normal Excel report. So using Power BI and other visualization and techniques, what we're hoping to do is highlight the changes, highlight stuff that we should sit there and talk about where it's not just, oh, does everything look good? It's, Hey, this system's doing something different. What are they doing? And finding out what actual systems are doing. And we've started creating some Power BI reports that really highlight those and save us tons of time on the back end because instead of sitting there filtering through a report of 2300 schools, here are the 10 schools that we identified with this crazy change.

00:22:26.43 AMY BITTERMAN: Mm-hmm.

00:22:26.65 ADAM CHURNEY: Let's talk about them. Is this something that we need to fix? Is this something that's going to happen more in the future? And just using visualization and the abilities with the data dashboard to help communicate not just to my teams and they're making sure the data's correct, going into EDPass, but also having reports for Dominique and other teams around our state to say, Hey, here's your data in a nice pretty fashion with colors. And then highlight the stuff that, here's stuff that you might want to know, because a lot of data, 95% of it in, you know, most cases is the basic case scenario. I always like to say my enrollment scenarios, most students come in day one in school A and leave in school A at the end of the year. Once you start getting to the next level of students that transfers and all this other stuff. Those small 5%, last 1%, et cetera.

00:23:17.12 AMY BITTERMAN: Mm-hmm.

00:23:17.41 ADAM CHURNEY: Are really where the interesting stuff happens. And the quicker you can get to those and start talking about those, that's where you get the final little bits of data quality resolved.

00:23:27.52 AMY BITTERMAN: Mm-hmm.

00:23:27.80 ADAM CHURNEY: And that's what with EDPass being able to iterate through stuff quickly, that's really helping there. Our hope is with our Power BI visualizations, we're going to get that before the end of the window and really understand our data to the next level and create knowledge and wisdom of the data that we're submitting to EDPass.

00:23:45.86 AMY BITTERMAN: Yes. And really use the data that you're collecting, be able to-- in Dominique's group, and then with the schools and the districts.

00:23:53.79 DOMINIQUE DONALDSON: Right.

00:23:54.17 AMY BITTERMAN: It sounds like providing them the data so they can really use it to think about what are we doing that's working well, what maybe isn't working as well, what changes do we need to make? So all of the stuff that you hope finally happens after you've kind of feel secure in your data quality.

00:24:10.91 DOMINIQUE DONALDSON: Yes. I absolutely love that because the power BI visualizations will provide the insights that LEAs need to really identify the positive impacts on what they are doing and identify opportunities for growth, all for student improvement.

00:24:32.55 AMY BITTERMAN: Great. Well, looking forward to hearing more. Maybe you both can come back and talk more about those reports once you've gotten them going and are using them with the state and districts, we'd love to hear more about them.

00:24:43.40 ADAM CHURNEY: Thank you for having us.

00:24:45.09 AMY BITTERMAN: Thank you both so much. Really appreciate it.

00:24:47.66 DOMINIQUE DONALDSON: Thank you so much

00:24:50.47 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org

View Details

00:00:07.34 >> Hey, it's Amy, and I am so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.76 >> Hello, and welcome to "A Date With Data." On this episode, we're going to be throwing a spotlight on 619 data with Candice Taylor, who is the Early Childhood Special Education 619 Supervisor, and Becky Palculict, who is the 619 Coordinator, both with the Mississippi Department of Education. Thank you both so much for being on the podcast. And I would love to get things started by having you each say just a little bit about your role at the Department. Candice, do you want to kick us off?

00:00:59.81 >> Absolutely. Thanks for having us, Amy. So I am Candice Taylor. I have been with the Mississippi Department of Education for 9 years. This is my 28th year in education. I've done all kinds of special education work in public schools over the years, and I've been ... I was 619 Coordinator for several years and then finally was able to grow our team, and now I have Becky.

00:01:23.44 >> Yes, Becky, what about you? Tell us about your role.

00:01:27.40 >> Well, again, thank you for having us. I am Dr. Becky Palculict. I am the 619 Coordinator. I took Candice's place, and she has been a great mentor to me. This is my 32nd year in education, all in special ed. I have enjoyed learning everything there is to learn about preschool education. And I ... My primarily ... My job is to provide technical assistance, answer questions or whatever for the districts within our state and to help them with whatever it is they need help with.

00:02:09.19 >> Okay, so all things 619. That is ...

00:02:12.00 >> Yes ...

00:02:12.57 >> That is you.

00:02:13.05 >> ... all things 619.

00:02:15.39 >> Great, so we have the right people on the call on this podcast episode, because that's what we're going to be digging into, is 619 world. And to kind of set the stage, can you start off by just telling us kind of the data story of your 619 indicators?

00:02:33.84 >> So I've always done this work in what I feel like are three buckets: indicator six, indicator seven and indicator 12.

00:02:40.66 >> Mm-hmm.

00:02:41.18 >> And we ... I've tried to do work on each one of those as we've gone along, and as far as our indicator six, we really ... We don't have universal pre-K in our state, so that's one of the things that we've really worked on a lot over the years, is trying to increase the number of seats for preschoolers, particularly in general education programs. That's what we really want, is to really help that LRE data.

00:03:07.84 >> Yep.

00:03:08.07 >> And one of the ways we've done that is offering what we call the blended program where we provide funding to school districts to have them start new classes that are gen ed classes but to make sure they include children with disabilities so that we can increase those opportunities for our children.

00:03:30.51 >> Mm-hmm.

00:03:31.04 >> As far as our indicator seven data, we have changed our process for how we collect that data over the last few years. We have gone from a single screener to the child outcome summary process, and so that's kind of been where that work has moved and is moving toward now.

00:03:50.87 >> Mm-hmm.

00:03:52.05 >> And as far as our indicator 12 work, we do joint trainings with our part C folks, and we really try to keep ongoing communication with them. And we're actually looking to increase that connection with them. We are about to hire a coordinator for our team that is specific to the CSPD, the Comprehensive System of Personnel Development.

00:04:16.02 >> Mm-hmm.

00:04:16.30 >> So we'll be able to have that person really work on those connections with part C, as well. Get that data up.

00:04:22.54 >> Yeah, that's wonderful, having especially someone that part of their position is focusing on that, because I think that tends to sometimes fall by the wayside, or it is kind of a little piece of what one person is responsible for and doesn't always get the attention that it needs. So that's exciting. Can you talk about some data quality challenges that you've encountered over the years? And what are some ways that you've tried to address them?

00:04:50.26 >> So one of the big things that has happened during my time doing this work is historically our state used a single screener to collect data for indicator seven. And when the time came for our contract to end with that provider, we decided to move to the Child Outcome Summary Process because so many states and territories use it, both in part B and part C.

00:05:15.67 >> Mm-hmm.

00:05:16.46 >> And what we really ... We really felt like that single screener was only giving us a snapshot of what the child could do.

00:05:25.02 >> Mm-hmm.

00:05:25.35 >> And we really felt like if we moved to that COS process, that would give us more of a whole picture of the child. So we're still a little early yet in the process to see what that data looks like, but we anticipate a real increase in the quality of our data since it's not just that single screener that's determining the child's outcomes but really letting the IEP committee decide and talk about the progress that the child has made. So that's been a big one for us. And, Becky, you've also worked on some data things for indicator 12, as well. Do you want to talk about that?

00:06:04.84 >> With indicator 12, the data team and I have met, and we'd meet twice a month about whatever data it is that we need to be speaking about.

00:06:14.93 >> Mm-hmm.

00:06:15.70 >> We have revamped how the districts and part C ... part B and part C talk. We require the districts to contact part C monthly and then provide us with a report to help us know that there is a communication between them now, and we take that data and look at it to see what it is that's needed. And I believe it's really helped with knowing what the transitioning of children from part C to part B.

00:06:52.06 >> We're also using technology to try to increase our communication, as well. We changed our notification system, and so now in addition to the our technology folks who are in a different office than us, Becky and I also receive a notification when we receive data from the part C side because our systems are not connected. We hope someday they will be, but historically and right now they've not been. And so receiving that notification lets us know that that communication is still flowing.

00:07:21.75 >> Great, and that's a step in the right direction, it seems, to getting it all fully connected.

00:07:26.11 >> Absolutely.

00:07:26.45 >> And then once you get the notification, does that go to the district, as well, or do you then have to pass it off to the district?

00:07:32.70 >> So our technology folks do that that are able to put that information in our student information system that we use for our state.

00:07:39.66 >> Okay.

00:07:39.75 >> And that way, districts get notified. But that lets us know that things are still moving along.

00:07:45.46 >> Yes, yes, and that ... Yeah. That seems probably not completely seamless at this point, but at least there is that process that is being followed, and it seems like things can happen fairly timely, hopefully, doing it that way. So we talk a lot about data culture and trying to build the capacity of the districts and SEA staff around data culture. What are some ways in Mississippi that you're really trying to strengthen the data culture?

00:08:17.47 >> Well, as Candice had mentioned, we have been meeting with the part B friends that we have. We provided joint training across the states and regional training with different districts and those that might need the training and provided assistance with them that way. Then we also provide technical assistance and monitoring to the different districts as they are trying to understand this new COS process that we're doing.

00:08:52.89 >> Mm-hmm. Mm-hmm.

00:08:53.78 >> Many of the districts are still ... Even though this is, I believe, our third year into the COS process, they're still trying to determine how they do it because they've had turnover in the people who were trained in it to begin with.

00:09:09.90 >> Yeah. Mm-hmm.

00:09:10.78 >> So we're retraining people, and it's getting there. They're learning, and they're understanding the process of how it's all supposed to work and come together cohesively.

00:09:22.40 >> Yeah, that's probably a challenge that goes without saying, is turnover.

00:09:25.68 >> Mm-hmm.

00:09:26.03 >> I know we're not the only ones in our state with that problem. We ... Other states are experiencing that, as well. It is hard to retain people in our school districts, and so even though we train and we train, we feel like every year we kind of start over with some folks, having to provide additional training to people who are new.

00:09:46.02 >> We actually have been out to many districts that have come together and gotten neighboring districts together, and I've provided a little small regional training and been there for them to ask those questions so that there's a lot more understanding, I believe. And then knowing that they can contact me ...

00:10:10.27 >> Mm-hmm.

00:10:11.01 >> ... and ask any question that they have has gone a long way because pretty much daily, I get an e-mail or a phone call that says, "Hey, can you help me with this?"

00:10:20.75 >> Mm-hmm.

00:10:21.05 >> And if I don't know, I'm sure I find the answer, and I get right back with them. So that technical assistance has really helped throughout the state with the ...

00:10:32.78 >> One of the things about being able to build our team that has been so great is that when it was just me, I had a very limited capacity as to being able to stay in touch with the data. And so one of the things that Becky and I have talked about from her very first day is that I really wanted that. I wanted her to get connected because we, Becky and I, work in the Office of Early Childhood. Our data folks, our data manager and all of our data team, they work in the Office of Special Education, and so we don't live together.

00:11:05.70 >> Mm-hmm.

00:11:05.91 >> We're not even in the same building.

00:11:07.34 >> Oh, boy.

00:11:07.78 >> And so one of the things I really wanted Becky to be able to do when she came on was to be able to coordinate more often. And so she's been able to have meetings with the data team and herself, just really trying to keep up with what's going on instead of waiting until data gets done yearly and then find out after the fact.

00:11:30.47 >> Yeah, that's one thing, just having worked with you all in Mississippi for a while. There definitely is a very strong relationship and connection, it seems like, between you all with 619 and part B data and the program side of things there, too, so that's a real plus, I think, when you have those tighter relationships, especially around the data. What are some plans you have for moving forward and continuing to improve the quality of your 619 data?

00:12:01.02 >> We have actually got more training throughout the districts planned. Also, I have ... Mississippi was chosen among four states to be part of a cohort to ... It's basically the trainer cohort, but to be able to use the COS process to make the IEPs better, being able to use those outcomes to write the goals so that they are more appropriate and to write the IEPs as smart IEPs for the children as they move into kindergarten and first grade. And so the plan is once the cohort is over, the team that has been picked to go and do the training, they come back and do regional trainings across the state with the districts so that they can learn how to use those exit outcome summaries and scores to write better IEPs that are more appropriate for the children as they move through their kindergarten and first-grade years and further education.

00:13:24.68 >> That's really exciting, because that's a great example of we collect all this data. Well, how are we using it? Sometimes we don't use it for a whole lot beyond kind of our compliance and the standard reporting that you need to do. But this is a great way to show how you're really going to use that data. To make changes and improvements, so ...

00:13:46.11 >> Right. Make it cohesive. Move through the pre-K, kindergarten, first-grade years and beyond. I'm really excited about it. We all are.

00:13:57.32 >> And that's one of the things that we continue to struggle with, we feel like our districts kind of struggle with, is it's not about collecting data to check off a box to submit to the state so that we can submit to The Feds. It's about seeing where our children are and how to provide better services for them, more appropriate services, to increase their outcomes.

00:14:20.73 >> Right.

00:14:20.83 >> And that's the big thing that we're trying to help districts understand. It's not just about those numbers. It's about boys and girls and helping them have the best outcomes they possibly can.

00:14:31.57 >> Yeah, absolutely. Well, thank you both so much. Learned a lot about what's going on around 619 data, and sounds like some really wonderful work happening and coming up soon, so thank you both so much for sharing your story.

00:14:49.96 >> Thank you for having us.

00:14:51.09 >> Yes, thank you.

00:14:53.84 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

Episode Transcript ###

00:00:01.52 >> You're listening to "A Date With Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.68 >> Welcome to "A Date With Data." Kim Murray, who is the Director Special Education, and Jennifer Nicosia, who is the 619 Coordinator with the New Jersey Department of Education, are here to share with us New Jersey's 619 data journey. So glad to have both of you here today. Can you tell us a little bit about your background and your role with New Jersey?

00:00:48.21 >> I've been the Director of the Office of Special Education for 2 years in February, so it's a rather new role for me, and it's exciting to be in it. But I have been with the Department in the Office of Special Ed for about 15 years, mostly on the compliance side monitoring complaint investigation and everything that comes with policy and procedure.

00:01:16.41 >> Great. Thank you. Jennifer?

00:01:18.77 >> Good morning. My name is Jennifer Nicosia, and I'm the 619 Coordinator for New Jersey, and I work with Kim in the Office of Special Education preschool. And what my position really means is that I support preschoolers ages 3 to 5 across the state who have individualized education programs. And on a day-to-day basis, that entails providing technical assistance and professional development and working with different organizations and agencies to support those students. I also have a background. I won't say how long, but let's say 20-plus years I was a special education teacher, and I worked for many years in public school system in special ed and preschool.

00:02:08.59 >> Great. Thank you both so much. So let's start off with the story of your 619 data, and, Kim, since you've been with the Special Education Office in the New Jersey Department of Education for such a long time, I'm wondering if you can just talk about some of the data quality challenges that have come up over the years and ways that you've all gone about addressing them.

00:02:32.28 >> Sure. I'd be happy to. With our 619 data, for the longest time, each program office had one data person who was in charge solely of collecting the data, verifying the data as valid and reliable and reporting the data. And that led to some really siloed work because that person was the only one who really had an in-depth understanding of the data and our data collection processes. So it was very hard to improve that when it was held with one person. In 2016, the Department created the Office of Fiscal and Data Services in our division, which, now we have a director who understands data, who loves data. She has a team working on the data, so we were able to pull together some more robust processes around data collection, data reporting, our data dashboards. So that piece has been great, and we're building on that work and the collaboration between the offices in terms of how we use the data and how they are collecting and analyzing the data. One of the issues that we have had with our LEA that they were struggling with submitting accurate data whether it was a lack of training. The staff member maybe who was inputting the data maybe wasn't someone familiar with the data, data system, so that was an issue. So we were seeing a lot of gaps in the data, incomplete data sets. Not every student would have a complete ... All the fields were completed, which means our data is not valid and reliable. So we've done a lot of training with specific districts. We've identified those who have the most issue with accurate data and making sure that they're getting some personal training and technical assistance to improve the quality of their data. And in terms of performance data, when we in general talk about Indicator 7, we had used the Battelle as our basis ...

00:04:55.00 >> Mm-hmm.

00:04:55.58 >> ... for our performance data. And what we were seeing is not many districts were using it, so the data set was getting smaller and smaller, and it wasn't really capturing what was happening in the preschool world for us. So now we've transferred over to the CODs, and we're building that up, which is great. And Jen has done an amazing job with ramping that up and improving the quality of data that we're collecting. Then we've also seen that the LEAs don't really have a connection to the data. They don't see the story the data is telling. So they submit it, and it's kind of, "We're done here. We've submitted it. We're not really analyzing it." So we've been doing a lot with building out data dashboards, making the data more accessible, more user-friendly, and then really explicitly stating the connections that the data has to program improvement and program development. And that's our goal, is to have the data really drive what's happening in terms of decision making and program improvement.

00:06:07.84 >> Yeah. That's a great strategy because if data are being collected for no real purpose or the LEAs can't see the purpose, then why are they going to put in so much effort to make sure it's of high quality? But if they see what it's being used for, how they can use it, how it can benefit them and their own students and families and teachers, then that just gives them more incentive, obviously, to make sure their data are accurate and of high quality.

00:06:35.14 >> Yeah, exactly. That's our goal.

00:06:37.62 >> What are some ways that you're working on strengthening New Jersey's 619 data culture?

00:06:43.40 >> So I think Kim told such an accurate story, since I'm here about a little over a year, if you could believe that.

00:06:49.99 >> Mm-hmm.

00:06:50.75 >> So she definitely told the story, and so by the time I came along, a lot of these things were in place. And so I was able to work with, under Kim's leadership and with the new monitors and the fiscal office and some of our colleagues who we could attest to absolutely love data and help to explain the why. So once we did that and we worked closely with the LEAs, what we were able to do was, through coaching, through a lot of technical assistance and professional development, we were able to really start to strengthen New Jersey's 619 data culture. And what I like to say is, so we have Alex Pensiero who works with us, and I would be remiss not to mention her because what she did was she actually makes data not scary.

00:07:43.68 >> Mm-hmm.

00:07:44.10 >> And so she's enabling us to work with all the LEAs across New Jersey and to change it from sort of being fearful of data, and once you understand the why and you start to understand the importance of the connections between everything that we're doing for preschool environment and least restrictive environment and then outcomes, it really starts to change the mindset, and it starts to really improve the data culture to be much more positive, I think similar to what Kim said talking about the why, tailoring the professional development, and having relationships that are really very positive with the local education agencies and especially the leadership of each district that's submitting data has allowed us to really significantly impact the work that we're getting. So clearly our processes are becoming stronger, but just the nature of the mindset, being comfortable talking about the data, and what does it really mean? And being honest, having those honest, open conversations, I think. And so some of the very specific ways that we've been able to change the data culture is by really sharing it and having things similar to what Kim meant. We have a dashboard where schools can go ahead and look at their data. We can provide technical assistance where we go ahead together with an LEA and look at their data and sort of analyze it and then figure out what it means. We have a few projects going on with some external stakeholders based off of the 619 data. And so I think what this does is it changes the culture by sharing our vision about meeting the needs of all children in early childhood.

00:09:42.64 >> Getting that data out there, getting the exposure, the transparency, and I like what you said, making it so it's not as scary or scary at all by kind of holding their hands and really walking through it w them so that they can understand it and be able to explain it themselves, is so, so important to building that strong data culture. Great, and what about moving forward? Do you have plans, any changes, new initiatives that you'll be putting in place to really continue on this journey of improving the quality of your 619 data?

00:10:19.93 >> Data quality is a priority for us, and it's ongoing, right? We're always building, seeing how we can do better, seeing how we can be more accessible. So in our office, everything we do is built on two foundational beliefs: equity and inclusion. And we don't make decisions without the data to back it up and to drive our initiatives forward. So when we're talking about inclusion in preschool, which is something that Jen and I hold very dear, and how we can improve our outcomes, we have external partners that we're working on around this, but the data is driving that.

00:11:02.09 >> Mm-hmm.

00:11:02.85 >> And that starts the conversation. When we meet with an LEA, here is your data. Let's talk about it. This is what it means. This is what we're seeing. These are your trends. So everything we do is around ... starts with a data review.

00:11:20.39 >> Mm-hmm.

00:11:21.02 >> And we imagine that will continue to do the same, right? That will continue to grow. We continue to work on data quality, whether it's Jennifer working with the system where we collect the Indicator 7 data. She's constantly refining that as we see where an LEA is struggling or they've identified sort of a work-around that we didn't know existed. We go in, and we fix that. So we make that system more robust. We're working on business rules to make sure that LEAs are submitting quality data and complete data sets. We work with our Office of Fiscal and Data Services to see how we can reimagine this data, how we can tell the story with the data and not just make it accessible for the LEAs. That's just one piece of our audience. But we have families who are interested in this and our external stakeholders and our advocacy groups. We see that sort of collective impact, everyone being on the same page and speaking the same story about the same data ... the data in the same way. That's how we can leverage our work, and that's how we can maximize our outcome. So we really are trying to develop this common language and common understanding around our data.

00:12:47.60 >> Yeah. I love that so much, just how you're being so thoughtful and intentional of thinking even just beyond the SEA and the districts but how critical it is that families and community and other stakeholders need to be at the table and understanding this data, too, and really part of the conversation.

00:13:09.26 >> And I think I would be remiss if I didn't say that we continue to engage in really nice conversations and professional development and training from IDC. We always ask for help on how we can improve our data, how we can make it better. We're very grateful for that because it has helped drive a lot of the improvements that we have seen in our data.

00:13:32.64 >> Great. Well, thank you, really appreciate that. Well, thank you both so much for spending some time chatting with me, doing some really great work in New Jersey, and love to hear what's happening next and what changes and how this is all going moving forward. So thank you for your time.

00:13:53.53 >> Thank you so much for having us.

00:13:55.74 >> Thank you.

00:13:58.30 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the Podcast page on the IDC website at IDEAdata.org.

View Details

00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Data with Data" is brought to you by the IDEA Data Center.

00:00:24.78 >> Welcome to "A Date with Data." Authentic and broad stakeholder engagement is required as part of the state performance plan annual performance report, or SPP-APR. And on this episode, I am joined by Susan Bineham, who is a manager in the Special Population Strategic Supports and Reporting Division of the Texas Education Agency. Susan is going to share with us how in Texas they've been able to continuously and meaningfully engage stakeholders in their IDEA data. Welcome, Susan. Thank you so much for joining us.

00:00:59.17 >> Well, thank you so much for having me.

00:01:01.70 >> Great. To get us started, can you just say a little bit about your role, and how long you've been with TEA?

00:01:08.86 >> I started with TEA on August the 1st of 2018. So Texas rolled out a brand new, big monitoring division and program, and I was originally part of that group that created our new monitoring program. And then in 2020, I became the SPP-APR coordinator.

00:01:32.55 >> Great. And you're still the SPP-APR coordinator currently?

00:01:36.25 >> Yes. Yeah.

00:01:36.45 >> Okay. Wonderful. All right. So we're talking about stakeholders, and I know in Texas, you have several different groups that you convene. Can you just start off by telling us who is part of these groups, and what is the purpose for each of the different groups that you have?

00:01:56.17 >> Sure. So first we have our continuing advisory committee, or our CAC -- that's the required committee we have; 17 governor-appointed members from around the state. They represent parents, general education teachers, special education teachers, consumers, special education liaisons. Most of them are individuals with disabilities, or parents of children with disabilities. They meet quarterly at a minimum. Agendas are posted publicly, and public comment is encouraged for this group. Their biographies and meeting minutes and recordings are all posted on the Texas Education Agency website.

00:02:45.48 They advise TEA on standards related to significant disproportionality, and they are required by statute to submit a report to the legislature bi-annually, which recommended changes to state law, agency rules related to special education, and the meeting date agenda and minutes are all published on the TEA website. We also have education service centers. So Texas is divided into 20 regional EFCs, we call them.

00:03:15.35 >> Mm-hmm?

00:03:16.08 >> They're intended to be the first point of contact for the local education agencies, and they are non-regulatory, they provide technical assistance and professional development for the LEAs. So we meet with -- each one of those centers have a special education director; they're not just special education centers. They service LEAs from pretty much anything, any capacity.

00:03:40.61 >> Mm-hmm?

00:03:41.87 >> We meet with their special education directors monthly to discuss new initiatives. But their role might be in providing training, how to support LEAs. They're really the first point of contact for LEAs, and because they're regional, if it's a real rural, small area, it has a different perspective than maybe people from the Houston, Dallas or San Antonio areas. So they provide very meaningful, good feedback to us on how we can support them better, and then they can support schools.

00:04:18.98 We also have a special education director's panel. So each of those 20 service centers, they recommend a special education director from a local education agency. So that is a select group of special education directors who work with TEA throughout the year. As I said, they're nominated to participate by their regional education service center. Their purpose -- it's a closed group -- their purpose is to provide feedback and input on initiatives and projects related to special education, including the SPP-APR. Presenting this panel provides an opportunity to capture the current needs in the field from the perspective of a special education director.

00:05:12.10 Additionally, this panel allows us the opportunity to gather stakeholder input and the time to collaborate with LEA special education directors that are currently in the field. We also have our Texas Continuous Improving Steering Committee, what we call "TCISC." This is the group that really works on a SPP-APR with us. They are an external work group tasked with advising topics as SPP-APR indicators, areas of slippage, Indicators 8 and 14 results in outreach, sampling plans, potential legal rule changes, legislative updates, state assessment participation, and our Indicator 17, our SSIP.

00:06:02.34 This group has about 15 people representing key perspective roles across diverse perspectives. The nature of this group represents parents, teachers, service providers, evaluation personnel, special education directors, campus administrators, districts, EFC's higher education institutions, advocacy, professional groups, other related state agencies, and other stakeholder groups whose mission include education of students with disabilities. So this group really is our in-the-weeds workgroup. We do target-setting with them, and they -- we ask them to commit to about a three-year period so that we have people that are really informed on the process. This group is also, like I said, close, we do not record these meetings, a real work group. We do trainings on SPP-APR each year and discuss potential changes. We discussed our sampling plan this past year, made some tweaks to it for SPP-8, and they really helped us with that. In fact, they're going to help us get the word out on 8 and 14. Some of those advocacy groups are going to help explain why not everyone will be participating in 8, because there's a sampling plan, but we do a census for 14. They're talking to us about putting things on their web pages to help the validity of these surveys. We contract out for those surveys, so some people want to make sure, hey, is this spam, is this okay? So those advocacy groups, which sometimes, you know, people, they can be our worst critics.

00:07:58.45 >> Mm-hmm.

00:07:59.98 >> But that's why they're in the room, because if we had them in on the front end, that's the best place to have them. They can advise us, they can talk to us about potential pitfalls, how to improve things. And even they initiated, hey, can we meet with you separately to help you with these two surveys? So some of these groups, they're really great work groups. People that participate in them want to be there, because their voice is really heard, and we take what they say and integrate it. And if we can't, we tell them why.

00:08:36.03 >> Mm-hmm.

00:08:36.33 >> Like, okay, "Well, this, we can't really do this, but how about we look at it from this angle?" So it's really productive, great work groups. We don't waste their time. We send them information prior to let them know what we're going to talk about, so they can come in with some questions already ready for us, also, for each group.

00:08:54.49 >> Yeah. Wow. So it sounds like you are doing so much in the state, bringing in so many diverse stakeholders, such a broad group. And you're doing just a wonderful job, that's so exciting. And I know other states out there in particular would really like to hear if there are certain strategies you're using that have built stakeholders' capacity, so that you're not every year having to start over again with explaining the indicators, and building that base for them, that they have a lot of that information going in, and being able to sustain the groups, like you said, asking for that three-year commitment. Yeah, what are just some ways that you've really been able to build a capacity and keep them engaged?

00:09:40.58 >> Well, some of it is what you'd typically expect. We present with the audience in mind. So we might have the same presentation for each group, but we're going to tweak it, or adjust it, given who's in the room, or who's in the Zoom.

00:09:56.77 >> Mm-hmm.

00:09:57.45 >> Our special education directors may or may not know some of this information. Our advocacy groups may know more than our special education directors sometimes, because they've been doing it a long time. And parents are coming from the parent perspective. So when we're presenting to a diverse group, we really try to keep all of those points I mind so we can -- people can understand and provide good feedback.

00:10:21.79 >> Yeah, you could tailor it.

00:10:23.59 >> Yeah. With that TCISC group I was talking about, the one that really works with our SPP-APR, we do presentations and training about each indicator, especially when we're setting targets. Even if they've been in the group for a while, because sometimes there's a little tweak to it. But they don't do this every day.

00:10:45.68 >> Yeah.

00:10:46.10 >> So each year, we'll do a little refresh reminder, go through the indicators where the data comes from, how it's used, why we set the targets where we set them. And they help with the target-setting. So they're thinking about things from their -- advocacy groups are thinking about things from the lens of families. So they, again, as I said earlier, they provide input and outreach information for SPPI 8 and 14 surveys.

00:11:14.81 >> Mm-hmm.

00:11:15.22 >> And they've had some really great input on some changes we've made with the digital world and people using their phones, they've provided some really great information for us on getting that response rate to move up a little bit.

00:11:32.23 >> Mm-hmm.

00:11:33.01 >> We also are a continuing advisory committee. They go beyond target setting, that's the regulatory group. Those meetings are recorded. They're posted on our website, the agendas are posted. We encourage public comment for those meetings. So that's the real regulatory meeting, I guess I should call it. The EFC directors, the service center directors, we talk to them about how their data from each of their regions impacts the SPP-APR.

00:12:10.43 >> Mmm.

00:12:10.89 >> So as I was saying earlier, some are real rural, and some are very urban. So we parse out the data to look at the difference between rural and urban, what kind of supports we need to help provide for them. We do presentations on individual indicators, the entire SPP process. And beyond the SPP-APR, we present on our RDA websites any new initiatives that we have coming out. So they don't advise us only on SPP-APR, we talk to them about everything; "Hey, we want to roll this out, what do you think?" And then people from rural committee can tell us, "Hey, we need this," or from urban, say, "We could do it like that." But that's -- we put pretty much everything in front of our EFC directors, because they support statewide.

00:13:04.13 Our special education director's panel, we went through this year our theory of action for our SSIP, and how the data from each LEA impacts the SPP-APR. So we really went through kind of the flow of what we provide, how it goes to EFCs, how it goes to LEA, how it goes to the teachers, the classroom, all the way down to the student level, because the SPP-APR seems kind of nebulous, it's this data-gathering, and we're writing this report, so when we really connected the dots for them, they can see, oh, I understand a little bit better now how this can actually support the students.

00:13:47.23 And there is a high turnover in special education directors, I'm sure all states are dealing with some of that. So this panel is our specifically-selected directors, but there's still some turnover there, so we do repeat trainings. But again, since they don't do this every day, they don't live and breathe the SPP-APR, any training, they're actually much appreciative. And they always say, "Oh, I learned something new this time" --

00:14:16.33 >> Yeah.

00:14:16.43 >> -- or something got tweaked a little bit so they can learn a little bit more about what we mean by significant disproportionality, and those kinds of things.

00:14:26.33 Our TCISC group, we do ask those members to commit to a three-year term. Those meetings are quarterly, but we frequently will give them, like, homework before the next meeting.

00:14:39.54 >> Mm-hmm?

00:14:40.25 >> So we'll send an email out saying, hey, meeting's coming up, these are going to be some of our topics, and give them some information so that when they get to the meeting, we can get right to work.

00:14:52.49 >> Yeah. I was going to ask, like, what's an example of homework that you might have them do?

00:14:57.75 >> We did -- when we were going through our sampling plan, we gave them kind of a draft of our sampling plan for 8 to look at. And it was pretty comprehensive, so it was a lot. We wanted to at least look at it. When we were doing target setting for this new six-year cycle, we had them kind of fill out a questionnaire so we could divide them into some groups. So we divided the SPP-APR kind of by indicators that would go together naturally --

00:15:30.65 >> Yeah.

00:15:31.25 >> -- and then send the information. And they met outside of our group in between our quarterly meetings to work on target setting. They called and met with some of us separately, but we only had the four quarterly meetings scheduled, and then they had separate work groups when we did all that target setting work. We also worked with them -- we'd do trainings and presentations for each individual indicator. We review the targets with them each year to say, these are the targets we're going through. And we also call them for our slippage reasons. So when we got all of our assessment data in January this year, we had a meeting already on the books with them to talk about potential slippage, and talked about what we felt the reasons for slippage were. And then they provided additional reasons. And we included all of that in the SPP-APR, because as I said, this is a very diverse group, working with parents and advocacy groups. So their input is really -- it really has a good pulse of what's going on out in the community.

00:16:41.70 >> Yeah. It's very valued, and you show that. It's not a matter of just checking a box, but you're actually incorporating in very transparent ways what you're getting back from them.

00:16:52.65 >> Yes. And they ask us about it. And they can see it in the SPP-APR.

00:16:58.16 >> Yep.

00:16:58.60 >> We're, like, "Here's the comment you guys provided us." So if -- when we do that, it's also very helpful, and it validates their work.

00:17:06.58 >> Great. Is there a highlight or a story, or something that you're really proud of in particular related to the stakeholder engagement you have going on?

00:17:18.24 >> We have some other stuff that goes on that I'm happy to talk about, because it's the most exciting thing.

00:17:24.76 >> Mm-hmm?

00:17:27.05 >> What I'm most proud of goes beyond what Texas does for the SPP-APR and target setting. We have a family engagement and outreach coordinator who manages our SPEDTex website. This website is available to anyone across the state. Parents can participate in trainings and focus groups. Has many resources provided in English and Spanish. Parents can create an account if they choose to. They can enter their child's IEP, the date of their last IEP, and it will send them reminders saying, "Hey, this is coming up, here's some things you might want to do." It has information on procedural safeguards, any of that. So that website is very interactive. And it's specifically targeted at parents and families --

00:18:22.22 >> Mm-hmm?

00:18:22.69 >> -- to help them understand processes. And they do webinars -- whenever they do a webinar, it's also done simultaneously in Spanish, so that both groups get the same information at the same time; it's not just translated into Spanish. They present in Spanish.

00:18:40.38 >> Hmm.

00:18:41.05 >> So I think that's a really positive effort to include our Spanish speakers across Texas. We're a large state with a lot of different languages.

00:18:52.31 >> Yeah.

00:18:53.34 >> Our Dispute Resolution Process webpage has information on special education complaints, but also resources for special education dispute resolution. And again, this website is available in English, Spanish, Chinese, Vietnamese, and Arabic. So we work really hard to not just provide things in English, we want to hit those top languages across the state as best we can.

00:19:23.28 Also, when they're participating in the cyclical monitoring, we had a stakeholder survey for families that have students with disabilities. So if your school district, or LEA, is engaging with us for their cyclical monitoring, we send out this survey. The questions follow three constructs; engagement, like opportunities for staff to collaborate, or parents to collaborate, or how schools provide information out. So there's engagement, and then understanding how data is used when creating IEPs, importance of inclusion, and also competency -- so, areas they can improve. So it's engagement, understanding and competency. So the questions are just general questions, but the answers give a good understanding of this. Do you meet regularly on this, so we can measure a little bit, the best we can. It's kind of like a consumer survey that you would do.

00:20:28.65 >> Mm-hmm.

00:20:29.05 >> And again, these surveys are offered in Spanish, Vietnamese, Filipino, Chinese, Burmese and Arabic. And the way we set them up along constructs, we can run the data mathematically to see how they scored. We don't have to worry about translating all the information back, we can have each answer numbered and run understanding the engagement and competency through that.

00:20:57.19 So all that's pretty exciting. I really like that we work hard to provide different languages, some groups, some supports for parents. We're a big state, we do a lot with Zoom. So some of these stakeholder groups, we're able to reach the far rural areas, because they can participate by Zoom, travel isn't required, because that really hampers people's ability to participate if they have to come to Austin from El Paso. That's a couple of days. They're going to have to fly in, spend the night, go to your meeting -- it's a big ask, asking people to participate in person.

00:21:38.34 >> Yep. Yeah. As much as, I think we, yeah, are all kind of Zoomed out. But it's been a great tool for sure in terms of being able to involve a lot more folks to participate than otherwise might have been able to.

00:21:56.55 >> Absolutely. We definitely have more participation, more regular participation.

00:22:02.06 >> Yeah.

00:22:02.73 >> People can take a couple of hours out of their day pretty easily, rather than having to commit a whole weekend or something, to come into town.

00:22:12.94 >> Yeah. Well, you have so much wonderful activities and initiatives going on. Do you have plans for anything coming up? Any changes you're going to be making? Any new areas you'll be focusing on?

00:22:29.47 >> There are some things that our technical assistant and professional development groups are working on, and some contracts, so they will be touching base with our EFCs and our special education director groups. And we also talked to our TCISC about it, because we want their input on how we can message out to parents, hey, this is stuff that's going on, we want your input also. Or does your student engage in any of these activities? So any time we push out new initiatives, or things we want to change or tweak, we really talk to these groups, because they can also point out to us any unintended consequences, like "Oh, this project's working really great, why are you changing it?" Or, "Just do this little tweak, and this will be great," or, "We can really get a lot of teachers trained if we had this or that." So it's a lot of open back and forth conversations with our professional development groups, our technical assistance groups. And usually, I'm included in that because it affects the SPP-APR.

00:23:35.05 >> Yeah.

00:23:35.16 >> So we work really hard to show those connections, too.

00:23:39.07 >> Absolutely.

00:23:39.70 >> Those LEAs, with all our new special education directors that aren't familiar with the SPP-APR, they can see how, when they look at our theory of action, you can see the direct line from the state all the way down to the student, so it's very helpful. And like I said earlier, we do periodic trainings or reminders about that process also.

00:24:02.62 >> And that's so important to be able to tell the story so that those at the local level see how it directly impacts them and their students, and how their data feed into the larger region and state. But it all matters, and they can kind of see where they fall in it.

00:24:21.27 >> Exactly.

00:24:22.22 >> Well, thank you, Susan, so much. This was such a wonderful conversation. Learned a lot about what strategies you have, and the work you have going on with stakeholders. And hopefully other states will maybe pick up some tips. And we know this is such a big area that a lot of states struggle with in terms of sustaining groups, and finding diverse participants. But it sounds like you all are doing such a fantastic job, and it was so wonderful to hear all about it.

00:24:50.46 >> Well, I really thank you so much for inviting me. And I appreciate the time.

00:24:55.48 [MUSIC]

00:24:56.68 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or, connect with us via the podcast page on the IDC website, at ideadata.org.

View Details

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.73 >> Hello, welcome to an episode of "A Date with Data." Now that states have submitted their federal fiscal year '22 SPP/APRs and hopefully have taken a much needed break, it is time to start thinking about the next SPP/APR cycle, especially because there are changes coming. On this episode we have two of IDC's TA providers, Rachel Wilkinson and Nancy Johnson, who are going to talk to us about what those changes are, what are some implications that states might want to be thinking about and some ways that states can get ready for those changes. Hello, Rachel and Nancy. Thanks for joining. So a lot of states might be somewhat familiar with what these changes are, but some folks listening may not be really be aware, so maybe start off, if that's okay, Rachel, talking about what these changes are to the FFY 2023 submission that will be due in February 2025.

00:01:23.20 >> Sure, so the approved changes that are going to take effect in the FFY 2023 submission, there's really ... I think of it as three buckets of changes, so the first is a change to the introduction, and this will also be a change to the instructions for this upcoming submission for 2023. So OSEP has added language for the introduction instructions to include eight elements, and they provide them in detail, but they're part of the state's general supervision system. Some of this information might already be something that states are including in their introduction currently. Some of the items look similar to the additional language that was added to the template and the instructions for the FFY 2022 submission. But there are some key differences that we did want to highlight so people ... It's good to get this kind of percolating in the mind to think about now because these are things you'll want to be able to easily address when you do get ready to start the submission process for 2023, which is not as far off as we always want it to be. Yeah, so the things that stood out to Nancy and I when we were looking at the changes, particularly to the introduction, were the requirement to include a description of how student files are selected for monitoring for indicators and for the verification of correction of noncompliance. Nancy and I review lots and lots and lots of drafts of the SPP/APR and kind of eat, drink, sleep, dream all things SPP/APR for a good chunk of the year. And we do notice that this is something we're not seeing often in the SPP/APRs we're reviewing, so it's something states are probably going to have to start adding in, particularly that verification of collection of noncompliance section. So sometimes there may just be comments that people have that say something to the effect of, "We verified through additional records that the noncompliance identified for X indicator has been corrected," but this is really asking for more detail of how those files are selected. And so being thoughtful and mindful of how you're going to respond to this as a state so that you have kind of clear processes in place, I think would be a really helpful thing to prepare for now rather than later right before the deadline is caving in. And then the other item was a description of how the state makes LEA determinations, and then that includes the criteria the state uses and the schedule for communicating the determinations to LEAs. I don't know that every state has that information in an accessible, quick way to write down. A lot of us use our institutional knowledge or people who maybe have notes documented here and there but maybe not in a cohesive description. So that's something to also be thinking about and keeping in mind as you prepare for the revisions to the introduction and these new components. So that really gets at the introduction piece, so that's sort of bucket one for the changes, so the second changes, the bucket that we're looking at, so to speak, is for Indicator 4. And these aren't massive changes, so these aren't things that may rock people's world, so to speak, or be super challenging, but they still are things that we're not seeing in our reviews called out consistently. So for Indicator 4, the requirement is going to be to now actually provide the numbers used in calculations, and that's based on LEAs that met n-n cell size requirements. So the state is going to have to define what those n-n cell size minimum requirements are as well as the rationale used for those minimum n-n cell sizes. That's explicitly called out in the new instructions. And then for option one, so if a state is conducting analyses of long-term suspension and expulsion data based on the state rates or the comparison of LEAs to each other across the state. Then the state would be expected to provide the actual state long-term suspension and expulsion rate. And then if they use option two, which is comparing students with and without disabilities within the same LEA who are experiencing those long-term suspensions and expulsions, if states are using a rate-difference methodology, like the difference in suspension and expulsion rate for students with disabilities compared to those without disabilities, then the expectation from OSEP is that the state will provide the actual ratio used, that difference, or their rate ratio used if that's how the state is conducting the analysis. So that information now explicitly needs to be called out and spelled out rather than maybe a more general description of what a methodology is without those nuances. And I think Nancy is going to tackle the big and third and final change that is on everyone's mind.

00:07:13.59 >> This is the one that I refer to as the biggie bucket, Indicator 18, and it is on everyone's mind. We get a lot of questions about that. People want to talk about Indicator 18 so they ... Most states, I believe, are aware of this change, and it focuses on the state's exercise of its general supervision responsibilities to monitor LEAs, and it must include findings from data collected through all components of the state's general-supervision system that are used to identify noncompliance. So I'm going to kind of go through these in short spurts here for a second. It can include information from the state-monitoring system or from the state's data system, dispute resolution, fiscal monitoring, for example, so really looking at all your different systems, your general-supervision systems, you would be pulling data about findings of noncompliance and then the verification of the correction of those findings ...

00:08:18.93 >> And, Nancy, that goes ...

00:08:19.64 >> ... for ...

00:08:19.79 >> That goes beyond the SPP/APR Indicators. Right? So it's not just ...

00:08:23.90 >> Yes.

00:08:24.13 >> ... thinking about ...

00:08:24.47 >> It is not just the indicators. It's looking at the indicators, findings within the indicators, but also findings in the other places related to the indicators. Now for FFY 2023, this will just pertain to the compliance indicators. However, if, for example, you have findings for, let's say, Indicator 11, then you would also need to look at findings related to the evaluation of students to determine eligibility that you might have found in some other system within your state. Maybe through complaints, you had findings of noncompliance related to that same thing. There is also the related-requirements document that states would want to look at and ensure that they're addressing any of those things related to Indicator 11, as an example. So while it pertains just to the compliance indicators, it is more broadly looking at all components of your general supervision where you might have identified a finding of noncompliance. So along with this, the baseline for FFY 2023 is expected to be FFY 2023, and states will have to provide rationale if they use a different baseline year for this indicator. And because this is a compliance indicator, targets are expected or required to be 100 percent. Again, you're going to use your related-requirements document to ensure that all of the IDEA regulations or citations associated with each of the compliance indicators for this FFY 2023 year are being reported, anything related to that, any findings you make. And it is based on, and this ... Sometimes states get confused about dates, but it is based on findings that were made in FFY 2022 and the status of the verification of the correction of the individual and systemic compliance within 1 year of when those findings were made or when LEAs were notified of those findings. So it is data that states are looking at right now in terms of the verification of correction of findings that they're making currently in this reporting year that were based on findings from the FFY 2022 year.

00:11:06.22 >> Okay, so it's noncompliance from the previous year that currently or already states should have issued findings already but should get districts in the process of verifying and correcting.

00:11:22.07 >> Correcting, and the state ...

00:11:22.93 >> And the state verifying.

00:11:23.70 >> ... and I'm going to emphasize, again, that word verification because it's not just the districts correcting the noncompliance. It's that the state has verified, used some methodology to verify that those corrections were actually completed and are acceptable corrections, that they actually corrected that individual findings and any systemic compliance. So it's important that they understand that they have to do that verification process within that year's timeline.

00:12:01.26 >> All right. So now that we've talked through the changes, what are some recommendations for how states can really prepare for these changes? Nancy, do you want to mention a few?

00:12:12.35 >> Sure. I'll be happy to start. First of all, I would recommend based on Indicator 18, which is the biggie bucket, to dig into the related- requirements document and make sure that all of their staff offices department and any staff responsible really understand any of those requirements and are aware of those requirements as it relates to the components, and that they need to provide this information to whoever is then completing Indicator 18 so that they need to be able to provide any information about the findings of noncompliance for FFY 2022 and then the verification of the correction of noncompliance so that whoever is developing, there will be a table in the tool that has certain requirements to complete all of this information. And it's got to be a joint effort. One person cannot do all of this, and it's different departments. Different people are working on different components that all may impact what those findings are.

00:13:24.76 >> Mm-hmm.

00:13:25.22 >> So digging into those related requirements, I would say, is the first thing I would recommend people do. And then I would also make sure that you establish with your staff a common understanding of the verification of correction of noncompliance processes. That it is not just the district correcting the noncompliance, it's the state verifying and how they verified that the district corrected the noncompliance. We're also in the process, or I would recommend that districts, if they haven't already, create some kind of a tool or process for how they track all of their information for Indicator 18 because it is a lot of data they're going to have to submit. And the sooner they start collecting that information and continue to add to it throughout the year until they're ready to submit would be helpful. And then maybe meeting with your team just to ensure that all the components for general supervision are clearly articulated and reflective of the work that is going on in a state. I know oftentimes when I meet with states, they'll be talking about work they've done with stakeholder engagement or other general-supervision requirements, but they haven't documented it or haven't included it in their SPP/APR, and they really didn't think about including it. But yet they're doing it, so they really need a way to document that and report it in their FFY 2023.

00:15:02.06 >> Yeah. You want to get credit for all that good work that you're doing.

00:15:04.87 >> Yes. You do want credit for that good work you're doing, absolutely.

00:15:09.60 >> We know states are getting into this and trying to figure out what needs they might have, as is IDC, how we can best support states with these changes, so what are some tools, resources, technical-assistant services that are out there and available or maybe things that we're working on that are going to be in place to help states related to implementing these changes?

00:15:34.44 >> When you mention this question was resources that exist currently, so the data-processes documentation is really valuable for making sure that states are capturing all the relevant information for the indicators, so thinking about the introduction as well as the changes for Indicator 4. This might be a good time for states who've done the data processes to revisit those and make sure that the information that's now required is reflected there for those data processes, and so everything is consistent. What's in their process documents matches what's in the SPP/APR because we know OSEP really wants that consistency across different reporting avenues, especially with the DMS visits that they're doing and looking at all that documentation. And for those of you who haven't done data-process documentation or have but maybe are thinking you could use some additional support, just a reminder that IDC will support these efforts and come and meet with your team, document information, facilitate conversation. So definitely reach out to your state liaison if this is something you're interested in doing, especially as you prepare for these changes or for monitoring visits or all that good stuff coming down the pike, so ... And then the last thing I can think of is that the Interactive Institutes that will be in Atlanta this year in June, so II 2024 will have some sessions that will be addressing these changes in the SPP/APR, so just a plug to take advantage of that opportunity to go to II '24 and learn about all sorts of things related to IDEA data, but in particular there will be a focus on Indicator 18 as well as some of these other changes, so ...

00:17:38.58 >> Thanks, and everyone will get a chance to meet Rachel and Nancy probably in-person and ask questions that you might have thought about during this podcast during the Interactive Institute. So definitely much more to come related to these changes, and do please reach out to your IDC state liaison with any questions or support that you might need. And thank you so much, Rachel and Nancy, for being on and giving us some insight into what's happening and what states can be thinking about, getting ready for it.

00:18:12.44 >> Thank you for having us, Amy. It was my pleasure to be a part of this, and we look forward to seeing everyone at II '24 and having the opportunity to discuss more of this with our state colleagues and each other, and thank you.

00:18:29.65 >> Thank you, everybody. This was great to talk more about this, and as Nancy said, we'll certainly be in touch with more information and hope to see you in-person at the different events that are going on across the country, including II '24.

00:18:45.63 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:07.34 >> Hey. It's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special-education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.56 >> Hi. Welcome to "A Date With Data." In early February, states submitted their federal fiscal year 2020-2022 SPP/APRs. IDC reviewed and provided feedback for many states' SPP/APRs, so we have a really unique perspective on overall patterns and trends, areas where states maybe are really doing well, others that seem to be more challenging, and maybe there is some room for improvement. So on this episode, I am joined by the two IDC TA providers who led the SPP/APR review effort, Rachel Wilkinson and Nancy Johnson. Thank you both so much for joining me. So let's start off by hearing about, what are some things that states are doing really well in their SPP/APRs? Rachel, do you want to go first?

00:01:14.98 >> Sure. So we were lucky enough this year ... I think we saw about 34 states, some version of their SPP/APR drafts, so it was great to really get a feel for how different states are tackling the SPP/APR, and for a lot of them, we saw improvements in the development of slippage statements. We know that that's been a challenging area because it really requires that digging into the data, and so it's less, "We need to fix this problem, and we're going to fix it in this way," but more in saying, "These are the reasons that the slippage occurred," so we've seen a lot of effort being made to really do that deep dive into the data, so that was really encouraging. Yeah, and then let's see here. I think some states were really good in describing their general supervision systems, so they provided more details about what those systems entailed. That might include things like their monitoring processes. That was a new example of different items to include in the general-supervision field in the instructions and then in the template this go-round, so we saw a lot more rich discussion about what states' general supervisions are. And then I think one thing with all the states we reviewed drafts for ... One thing that we were encouraged by is that they were getting them in really early, so people were submitting things in early December and some even earlier than that, which was great because that meant that they could take the feedback and really apply it and make sure that their drafts were high quality and compliant, and that was really encouraging to see, and I think the SPP/APR Summit has helped sort of spotlight the importance of working on this sooner rather than later.

00:03:15.54 >> Yeah, definitely. It's nice to have that summit occur when it does to kind of kick off in some ways. States probably have been working and should be on their SPPRs well before that, but the time where it really comes to putting pen to paper and really getting down into it and reminding states, of course, at that point too that we're available to review their SPP/APRs, and hearing that so many states took us up on that is ... It's really exciting.

00:03:41.89 >> Yeah, it was wonderful. We were thrilled.

00:03:44.03 >> Great. It's really promising to hear about all the ways that states are improving and doing well, and we know there are still some areas where states could use some improvement, and I'm wondering, Nancy, if you want to talk a little bit about what some of those are.

00:03:59.82 >> Sure, I'd be happy too, Amy. There is always room for improvement no matter how well we do things, and there are some ... a couple of areas that jump out at us that could use some improvement. One in particular was in the area of Indicator 4. OSEP had some very specific comments from states to address some requirements related to reasonable methodology with Indicator 4, and the state ... Some states really did not address those requirements with the comments from OSEP. They may be waiting for more directive from OSEP with regard to Indicator 4, and also, some states have let us know they had already issued their requirements regarding Indicator 4 earlier before they ever got their clarification or the OSEP letter in June, so they just kind of waited until maybe this year to take a look at it. We would suggest though that it would be beneficial for states to be thinking about their existing processes alongside stakeholders, including their stakeholders to ensure that children with disabilities are not being subjected to inappropriate removals that could prevent them from accessing instruction and which then would impact long-term outcomes for children to their adult lives, so that Indicator 4 area is a real ... not a concern to us but ... And we're waiting to also see how OSEP is going to respond to those requirements. Another area is in the area of stakeholder engagement. Some states did a nice job in that area, but many states provided more generic information regarding stakeholder engagement or referenced work they had done prior years, particularly prior to with that first submission of FFY2020, which is 4 years removed from that now, but it's worthwhile for states to be connecting with stakeholders regularly on an ongoing basis, and they should be addressing more than just setting new targets but also looking at their data analysis, their progress toward their targets, their improvement activities and their evaluations, and it was sometimes challenging for us to find information in there about what currently occurred this past reporting year related to stakeholder engagement. And then a third area that ... And this is one that's been kind of ongoing ... is in the area of corrections of noncompliance. Corrections in noncompliance is really about the state verifying [Indistinct] LEAs corrected any findings of noncompliance, and that verification of correction should be consistent with the OSEP QA 23-01 that came out last July and the requirements within. In particular related to correction of noncompliance, we saw and continued to see challenges with the correction of systemic noncompliance where it is unclear whether or not states really looking at additional records in addition to the findings of noncompliance that they initially made and then looking at then additional records that they were reviewing and when those reviews took place. We also saw that states sometimes still combine their individual child-specific noncompliance and their systemic noncompliance all together and just repeat it in both prompts when they really are two separate prompts for that information. And in some instances, we just saw some general boilerplate-type language about their monitoring and review process rather than specific language about what the state actually did to verify the correction of noncompliance, which is the OSEP requirement. So those are three kind of main areas that come to mind when we think about ways states can continue to improve their SPP/APR.

00:08:17.06 >> All right. Thanks, Nancy. And coming up in April is what's called the clarification period, so if there are things that come to OSEP's attention when they're reviewing the SPP/APRs, it's an opportunity to come back and ask questions. I think I kind of got that right, but I might be missing pieces of it. So maybe for newer state staff, Rachel, do you want to say any more about what the clarification period is for those who might not be familiar with it?

00:08:44.89 >> Sure, so this is an opportunity that OSEP offers. It's usually about 2 weeks in length where they'll have comments, feedback, sometimes clarifying questions, thus the name of the clarifications to ask about particular indicators, components of indicators and the introduction as well as other areas that they might look at, so it's important when you get those clarification notices from OSEP to dig into the different comments they've made, and all of those will be in that SPP/APR tool in the EMAPS data system, and then states develop responses, provide those to OSEP, and then based on those final responses, OSEP will provide any comments or questions, feedback that they might have as a result of their reviews of the SPP/APR, and then they'll review states' responses because each state will need to respond to those comments or questions, and then they will issue either additional required actions based on that feedback or any other comments that they think are appropriate, and then that information is what's finalized and reported in the final SPP/APR.

00:10:05.14 >> Okay. Thank you. And IDC is able and willing to review how states are addressing and responding to anything that comes back from OSEP during that clarification period as well, so just putting that out there for states.

00:10:21.25 >> Absolutely, and we would encourage you to make sure that you have at least someone there who can take notes or be an extra set of ears, so IDC state liaisons are great to be that resource, and we really do encourage during this time frame that states take up the offer from OSEP to review the clarifications as a group and to have a call to do so because then you can ask questions and get feedback on things. Maybe you have a question about a comment that was provided that wasn't clear. These are the calls that you can get that clarification, and some states have asked for clarification and found that issues that had been flagged really weren't issues, and they didn't have to address them, so that meeting with OSEP and then having someone there to help you with taking notes or listening in could be really valuable.

00:11:16.80 >> Great, so that's, I think, a helpful tip for states when it comes to addressing OSEP's questions and feedback. Are there other tips that you all have to share that might be helpful for states?

00:11:28.13 >> Well, I can think of one more, but I think Nancy's got a couple as well because she's worked with a lot of states on this.

00:11:34.40 >> Yes.

00:11:35.72 >> But one of my additional thoughts was just to come up with a strategic plan of how your group as a state is going to respond to the questions and clarifications. That helps everyone understand what the expectations are and how they're responsible for certain portions, so if you have an indicator lead who needs to answer a programmatic question, they're aware of what that clarification question is so that you can be strategic in your response as a state, or if there's a data-specific question, make sure you have your data manager, whomever put the data together for the indicator available and aware of the question so that they can give feedback, so again, kind of coming up with that plan internally so that the responses you as a state provide are coherent and reflective of all the expertise that might be needed.

00:12:31.26 >> Yes, and I do also have a couple of tips, and I would like to go back to the importance of the call with OSEP because sometimes states think or interpret the information that OSEP asked for and then learn during the call as they're talking about it that OSEP really meant something different than what the state thought they meant, so those clarification calls are very important. Along with that, it's important as you're getting ready for that call that you review all of OSEP's feedback carefully before you have the call, and I want to emphasize including the comments for the introduction section. I worked with a couple of states that had overlooked the introduction section and only focused on the indicators, but there are comments also in the introduction section. And then lastly, for the required ... Think about, for those required actions that OSEP is asking you to address next year, you want to ensure that you're documenting these and beginning to develop plans on how to address those actions, and if you thought about that before your clarification period, you could perhaps consider even ... or during the clarification period running those past OSEP during the call or asking them about them during the call as you're thinking about them so that that will help you know how to address those required actions in the next SPP/APR and just documenting any of that information.

00:14:14.01 >> Yeah, while it's still kind of fresh in everyone's minds and you have that chance with OSEP, yeah, might as well use it, and that's a great, great tip.

00:14:23.47 >> And then I would also emphasize having ... inviting your IDC state liaison to be on the call with you because they can be that extra set of ears and might hear something you didn't hear or also taking notes for you so you do have things documented.

00:14:40.05 >> Yeah. Wonderful. Well, thank you both so much. I picked up some great tips. Hopefully others listening did as well and really appreciate you being on the podcast.

00:14:54.11 >> Thanks, Amy. This was a pleasure.

00:14:56.45 >> Thank you, Amy. It was our pleasure to do this.

00:15:00.32 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:07.34 >> Hey. It's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.61 >> Welcome to another episode of "A Date With Data." I am joined by several former data managers who are now IDC technical assistance providers. Austin Ferrier, Kelley Blas and Kristen DeSalvatore were all data managers up until a few months ago, and they're going to share with us their unique experience of going from being a data manager to now working to build the capacity of data managers and other state staff to improve the quality of their IDEA data. Welcome to all three of you. And I think, to get things going, it would be great if each of you could just briefly introduce yourselves, say a little bit about how long you were in the role as a data manager and what you're doing now. And, Kelley, do you want to start us off?

00:01:09.05 >> Sure. Thank you, Amy. Again, my name is Kelley Blas, and I was a data manager at the Department of Public Instruction in North Carolina. I was there for 17 years as the data manager and the last few years as the SPP/APR coordinator, and I'm very happy to have joined IDC as a state liaison and technical assistance provider, just love working with states.

00:01:35.64 >> Great. Thanks, Kelley. Kristen, do you want to say hello?

00:01:39.35 >> Hello, everyone. I am Kristen DeSalvatore, and I was the data manager at the New York State Education Department for about 11 years. In that role, I also served as [Indistinct] coordinator. In my role at IDC now, I'm working to provide technical assistance and support to states and LEAs related to the collection and reporting of education data and especially the special ed IDA data.

00:02:08.19 >> Wonderful. Austin?

00:02:10.19 >> Yes, thank you for that introduction, Amy. Afternoon, everybody. My name is Austin Ferrier. I am the former IDEA Part B data manager for the Florida Department of Education. I was actually in that role for about a year and 6 or 7 months, and as I transitioned to IDC, I have been focusing on CA specialists and 618 data.

00:02:36.26 >> Great. Thank you. So it's wonderful to have all three of you, and I'm so excited to now be able to work with you on the IDC and technical-assistance side of things. I was fortunate enough to get to know all of you when you were data managers so really thrilled to have you now onboard with IDC. And to get things going, I'm wondering if you all could reflect a little bit on your time as a data manager, if there is advice that you might want to provide, if there are newer data managers. Austin, you were in that role much more recently, so it's a newer kind of experience for you, just thinking about things you would've maybe wanted to know as a new data manager. Kelley, do you want to kick things off?

00:03:21.54 >> Sure. So when I was reflecting on advice for new data managers, one of the first things that came to me was to utilize all of the technical-assistance resources that are available to you. I know that when I started 17 years ago, I wasn't really aware of what technical-assistance resources were available to me. Felt like there were times where I couldn't find exactly what I needed, but now in this day and age and especially with having IDC, there's a website that's super filled with resources. There are technical-assistance resources, such as state liaisons. I would definitely encourage folks, especially newer folks, to invite their state liaison to walk them through all of the resources that are available on the website. Also, the other thing is just having a plan in mind for what your year is going to look like around data collection, when it's being collected, when analysis needs to happen, when submissions are due and just having that calendar planned out that's kind of unique to your state. Not every state collects things at the same time, but we know that it's all kind of due at the ... It is due at the same time to OSEP, so having that calendar planned out for the year really helped me know when I was supposed to take leave if I wanted to take a vacation or make plans and when I was going to need to be really, really focused on my data-analysis activities.

00:04:50.06 >> Yeah. Kristen, do you want to give us some advice?

00:04:53.72 >> Sure, so I'm going to echo what Kelley said about the IDC resources and really advise folks to make the time to really investigate the IDC resources that are available to you and to reach out to your state liaison on a regular basis. Like Kelley, I didn't really realize the depth of support that is available and really wish that I had. I would also tell people to network and collaborate with others in your agency and really try to find allies in the Office of Special Ed or data shop that you can work with and count on to help you get the work done that you need to get done. And lastly, I would say don't be afraid of OSEP. They're not out to get you, that a lot of times folks feel like, "We're going to get dinged. We're going to have a big problem with OSEP." So keep in mind really that OSEP has rules and regulations that they must follow, right? So they are pretty scripted by what's in the IDEA law. And also keep in mind that we all have the same goals in mind, right? We want to improve the landscape for students with disabilities. That should really be the underpinning of all of the work that you're doing.

00:06:21.45 >> That's great to keep in mind and especially for newer folks who might be intimidated by OSEP, but know that they're there to support states in any way really that they can, so they're a resource and a partner. Austin, what advice do you have?

00:06:38.69 >> I do have to echo some of the sentiments that both Kelley and Kristen expressed, especially what Kristen mentioned about, regardless of what we do, it does all come back to the students. We might be looking at data timelines inundated with messages from OSEP and emails, but one of the biggest things that really clicked with me as a new data manager, that this all comes back to fate, providing free and public education for kids and those support. So each time you see a number in an Excel sheet, any type of visualization you make, just try to keep in mind that that's a real student, a real person and that we are trying to do our best to support those groups. Another thing I would really advise new data managers to do, if you get the chance, please, please, I highly advise to attend one of the IDC Interactive Institutes summits or any type of project or presentation that IDC puts on. I know when I first made my initial trip for the SPP/APR summit, it opened up a whole new world outside the lens of Florida in terms of the level of collaboration, the level of support available, and it just really helped encapsulate what we are trying to do in our positions and in our positions as Part B data managers in support of LEAs. If your state allows you, highly advise to take the trip to any of those Interactive Institutes or summits. And then just finally, just as Kelley was saying, IDC has data manager connection groups, data quality peer groups. The level of collaboration you can achieve in those specific safe spaces is amazing. I know the conversation ... I've had conversation with Kristen herself when we were both in that role in meetings where we were talking about some very sensitive topics, but that collaboration really allowed us to build our own state's capacity and just kind of build that knowledge base.

00:08:56.86 >> Yeah, so it sounds like what I'm hearing as a theme from all of you is really the collaboration, the relationship building because often in the state you might be a little isolated, the only one really doing a lot of the work that's kind of in your head and the importance of working across your SEA, and then also just reaching out and tapping other data managers is such an important piece of the role. So now that you've transitioned out of the role of being a data manager, Kristen, what's something that you're really going to miss about being a data manager?

00:09:33.63 >> I'm really going to miss the collaboration with my immediate colleagues in the data shop at NISED as well as with the staff in the Office of Special Education. As a data manager, I really was the bridge between the two separate offices in the department, and I worked very hard to bring the data and build the data literacy in the Office of Special Education and did reap a lot of rewards from that and made some good connections with colleagues, so I will definitely miss that part of it. I'll also miss working with the data, and as Austin said, it's numbers on a spreadsheet, but the work that a data manager does, does have the potential to influence directly things for kids, right? So you do have the ability to influence practices and policies that do have that potential to make a difference really even in an individual kid's life, so that to me is something I will miss.

00:10:50.73 >> Thanks. Austin, what about you?

00:10:53.98 >> I just have to echo kind of what Kristen was saying in terms of that level of collaboration, but I will say probably one of the biggest aspects I'll miss, especially at the SEA state level, was the interaction I had with LEA exceptional education directors. I was a phone call away with almost every special director in every district in Florida, and having that connection and having that bridge really felt like the distance between the state and the LEA was lessened, and the trust was strengthened, so knowing that they could call me at any moment if they had a question regarding their data specifically, even at the school level, just building that trust and those relationships and having that direct assistance and seeing that have a direct effect in real time, one of the biggest things I'll miss. Second biggest thing I'll miss, I worked with some amazing individuals at the state level who, even outside of their career and job, were still focused on community outreach programs, were still going to school-board meetings after work. The passion was there, and it was evident, and I fed off of that, and I'll truly miss some of the individuals I ... They were superstars, rock stars.

00:12:21.34 >> Mm-hmm. Yeah. And, Kelley, what things are you going to miss?

00:12:26.55 >> It's funny. We're all kind of missing the same thing, but definitely for me, I'll definitely miss the friends that I made at the Department of Public Instruction across the agency, but the main relationship that I'll miss is, after 17 years and being involved in building our state special education data system, you develop some really strong connections and supportive relationships with our LEAs, and again, they knew that they could contact me, and I would know a little trick to get their data to go in just right or what the workaround was for our system, and so those kinds of phone calls, they're always so gratifying because they know that they can call you, and you'll figure it out for them. So those things I'll miss and definitely, definitely the data. I am a data geek at heart, I think, and so being able to look at those big data sets and knowing the rules of our state and how everything's supposed to fit together and how to present that visually where it makes sense when we're talking to our districts about their data, those are the top three things, I think, for all three of us that I'll miss.

00:13:41.01 >> Mm-hmm. Yeah, lot of similarities. I think as you get more into the role, the states that you're working with will sort of become like your districts were, and they'll be calling you, and you'll be calling them and build up that same type of relationship, so I think you'll still see that in somewhat different ways. And we touched on this a little bit already, but are there things that you want to mention that you as a data manager wished you had known more specifically about IDC and what IDC does and the services and the other TA centers as well? Austin, is there anything you want to mention?

00:14:18.98 >> Yes, yes, definitely I want to re-emphasize the area of safe space that IDC creates. I really do want to just express and emphasize IDC is not a punitive organization. They are not looking to ding you on any of your data pieces. It's a holistic examination of state processes as a whole in a ... It's pure assistance, so always keep that in mind. Come at it with a positive attitude and come at it as if we know where you're coming from, and Kelley and Kristen, they will agree. We've been in their shoes. So just having that empathy and knowing that we do try to create a safe space, and I really do hope states understand that.

00:15:16.08 >> Yeah, absolutely. Kelley, what are some things that you want to mention about IDC or other TA centers?

00:15:25.34 >> I think one of the things that I didn't realize about IDC that I know now just from my own experience as a state liaison is the multitude of ways that IDC and other TA centers can really come in and support states when it comes to the work that they're doing around data. For example, I'll be going to a state next month and helping facilitate a stakeholder meeting around some changes that they're making to their Indicator 4 methodology, and I wish that I had known that those kinds of supports were available to me as a data manager and to our team in general at the Department of Public Instruction because we could've utilized those supports in a way that would've made those meetings just richer and so just understanding the true depth of support. And I'll say it again because I didn't ... I definitely didn't understand this, but safe space, safe space, safe space. I think for a long time I felt like if I shared too much information with our TA centers that maybe I was airing my state's dirty laundry, and I didn't want it to be known that we were doing things wrong. I just wanted to help correct it. But knowing that these TA centers are here specifically to support us in improvement efforts and making things ... making our data stronger, it changes the whole understanding of what technical assistance really is, and I just ... I had some misconceptions for sure.

00:17:06.18 >> Great. And, Kristen, anything you want to add?

00:17:11.04 >> So, Kelley and Austin, you did a great job of taking the words right out of my mouth. I do ... One thing that I really didn't understand was how important it was or is to get to the conferences and institutes that IDC offers and pays for folks to attend. In New York, we had a hard time with travel, and we were finally allowed to go to ii23, Interactive Institute 23, and it was just mind-blowing for us. We were like, "Oh, my goodness, we have really missed out on a lot of in-person technical assistance and information and the collaboration and the networking and that piece of it," so that is one thing that I would highly, highly recommend is that all data managers but especially new data managers really work hard to be able to attend the in-person events that are offered.

00:18:19.61 >> Yeah, between all the data centers, all the TA centers, there really is such a wealth of expertise and knowledge and resources that we need to make sure all the states and staff and especially newer staff are aware of and can utilize. So kind of looking forward now, we talk a lot at IDC about what it means to be a data quality influencer and how everybody is a data quality influencer in different ways. You all as data managers were definitely, of course, data quality influencers in that role. Can you talk a little bit about some things you're excited about in your new role in terms of being a data quality influencer? And, Kelley, do you want to start us off?

00:19:09.72 >> So my mind is always whirling on what I could've used during my time as a data manager, and now that I'm with IDC and I know that part of the focus at IDC is creating tools and resources to help data managers and states have better tools to analyze and display their data and make meaningful change in their daily work and efforts, it's really just exciting to me to be able to think about what I could've used and how I can present those ideas and potentially create these tools to assist LEAs, for example, significant disproportionality or the indicators, just tools that will help them analyze and display data better.

00:19:53.13 >> Thanks, and really who better than the three of you and other former data managers to come on and help put those ideas and dreams you might have had as a data manager like, "It would be great if this existed, but I just don't have the time or the capacity to create it," and now can really be devoted to that and helping other data managers create those for them. Kristen, tell me a little bit about what you're envisioning in your role as a data quality influencer with IDC.

00:20:25.65 >> Well, Amy, as you said, the state will kind of become our LEAs or district, so I am looking forward to working with the individual states to provide one-on-one technical assistance as well as being thoughtful about updating existing resources to reflect new guidance, new practices and new perspectives and create new resources, knowing, as Kelley said, what I thought I might ... would be super helpful when I was in that role as a data manager. I am really excited to now be on the other side of the aisle with the deep understanding that I have of what data managers are going through, what their workload is like, how much information there is for them to process and get right, right ... There is very high-stakes data, this IDEA data ... and how important the work is. So if there is one data role, I think, in a state agency that you want to be a data quality influencer, it is around the IDEA Special Education data, right? It's super high-stakes, and I'm just very excited to be on the train.

00:21:50.25 >> Great. Yeah, we need a whole army of data quality influencers for this IDEA data. So, Austin, what ... How do you see yourself being a data quality influencer now?

00:22:02.51 >> Yes, Amy. When I think about this question, the word empathy keeps popping in my head. We ... Kristen and Kelley, we've been into EMAPS. We've had to submit our SPP/APRs on February. We've seen those emails from our bosses' bosses asking for updates or asking for some type of information that you have to put together quickly and translate it into something they can digest quickly. Having that empathy and having gone through the data-submission process, it really helps me understand what state SEA data managers are going through, and just knowing those feelings that they have, those crunch-time deadlines and just navigating your OSEP guidelines and then your state, federal ... your state guidelines and state legislation, just being able to ... Having that in the back of my head when I have conversations with states, just making sure that they know that we've been in those positions before. We know how you feel, and we're going to try to the best of our ability to help you in your data submissions.

00:23:26.18 >> Yeah, that makes so much sense. It is such value that you bring having had those experiences and gone through everything they're going through, and, like you said, having the empathy, that really adds so much to what you all bring. So thank you all so much for sharing with me and all of us your experiences as a data manager and now transitioning into this new role, what that looks like and what you're looking forward to and so happy and thankful to have you on.

00:23:59.55 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.62 >> Hello. Welcome to "A Date With Data." On this episode, we are continuing our conversation with Leah Voorhies, who is the State Director Special Education, and LauraLee Gillespie, who's the Special Education Coordinator of the Utah Program Improvement Planning System, and they're both with the Utah State Board of Education. On the last episode, they started their story about their state's general supervision system, and on this episode, we are going to hear more about some of the challenges that you've experienced, how they've addressed them and some of the areas they're really proud of, so thanks for joining us. So what are some of the challenges that you've encountered, and how have you tackled them to make these improvements and put these great processes in place?

00:01:12.60 >> Well, I'll speak to a few challenges that I see. Bridging the gap between compliance and program improvement, again, we know from the Supreme Court that we look at benefit. We look at procedure, and we ... That's been the requirement in special ed for decades. But sometimes procedure gets a little more focus, and although it's important, we want to ensure that those procedures are actually leading to outcomes for students and program improvement for schools, and that, I think, is a huge challenge that I personally feel it's not ... It's easy to just look at a file and say, "Here's the issue," but it's much more difficult to look at a file and, again, look at it over time and really see what we're doing to improve outcomes for students, so I would say that, in talking to staff in LEAs, in talking to individuals here at the state and talking to parents, that that is probably the largest challenge, and so trying to bridge that gap between the importance of compliance, and I do have a place for compliance in my heart because of my legal background, but I ... But it is important to really understand the why behind it, why it's important and why I will say honestly there's lots of conversations I have with local education agencies in particular, helping them to see why it matters from the parent point of view, from the stakeholder point of view because that's kind of where my background is, and so that piece, I think, is one of the most challenging obstacles that I think we face in our processes. There's lots of logistical issues that are involved, but I would say if I was going to pick one thing, that would be it.

00:03:17.16 >> Mm, yeah, that is a big challenge for sure.

00:03:20.74 >> Leah, would you ...

00:03:24.01 >> I think another challenge we identified last year and the way we addressed it possibly unique ... We're still not sure if it made a difference or not, but when the monitoring team sits and reviews files, they give the teacher of that student, the file, an opportunity to sit with them, and so they talk through the process if the teacher is available and really listen to the teacher talk about what they were trying to accomplish for the student and help the teacher understand the compliance issues. And we recognized that, because we don't have a statewide IEP system in Utah ... There are 16 different IEPs, IEP systems, and then lots of LEAs use our model forms, just paper-pencil model forms, and we recognized that teachers struggle with understanding how to fill out the forms. Teachers struggle with understanding the why of filling out the forms. They struggle to understand the why related to the rule, but they also struggle like LauraLee was just talking about to understand the why of what the relationship to student outcomes is, and sometimes it doesn't really look like there is an outcome, so then they're just frustrated.

00:04:44.57 >> Yeah.

00:04:44.76 >> Right? So we have a statewide institute on special education law, basically a law conference, every year. We've been doing it for 35 years or something, and every summer we spend 2 days. We bring in attorneys from around the country to talk about important issues related to our state and to our general supervision system, and we did something different the first time in our, however, 35 years. LauraLee and her team did a session with almost 2,000 people who attended our conference. It did 3-hour session just going through a file with the whole state line by line by line. They made up a student. And they just had 2,000 people in the room and online that did a file review together so that she could have that conversation that she likes to have and her team likes to have with individual teachers with the whole state.

00:05:55.69 >> Wow.

00:05:56.55 >> And so that's one of the ways, one of the challenges that we found and that we've addressed, and the other big challenge, which we try the same way every other state tries, is that we have staff shortages. Our LEA directors are overwhelmed. Our teachers are overwhelmed. Our parent educators are coming and going. It's like a rotating door.

00:06:19.52 >> Mm-hmm.

00:06:20.49 >> And so there's always a need for professional learning. There's always a need for technical assistance. There's a need to provide emotional support to special-education providers, and so we have ... We've created synchronous, on-demand and asynchronous available at 2 o'clock in the morning professional learning and technical assistance so that our seriously overstretched educators and administrators can access information whenever wherever from us with a consistent message. We all say exactly the same thing, and to LauraLee's point about compliance supporting practice and practice supporting compliance, we have one message. We all say exactly the same thing, and you could get it from our mouths. You could get it from recordings from us and just to try to address that particular barrier.

00:07:30.21 >> Sounds like you've come up with a lot of unique and creative strategies to try to handle these challenges, and that's really fascinating and good for you all. And one last question, you mentioned the 2301 memo, the OSEP General Supervision Q and A that came out. Can you talk about how that has impacted, if at all, Utah, your system, kind of things you might be changing potentially, what kind of your reaction was to it?

00:08:03.34 >> Yeah, so my reaction was probably not as dramatic or concerned as some other state directors because we had already moved to a mostly cyclical monitoring cycle. We had already put in place a lot of the requirements in the memo. A couple that we needed to address, one, the transfer issue, ensuring correction once a student transfers across LEAs and across states, and then another one. We have a statewide education complaint hotline, and so anyone anywhere anytime can submit a complaint to the state board of education about anything that's happening in their school, and we have a system internally to be able to route the complaints related to special education to our Special Education Dispute Resolution team so that people that submit hotline complaints can get the information they need about IDEA dispute resolution and submit IDEA complaints that way or request mediation or a due-process hearing. So one of the things that we had to put in place was a process to move from that complaint-hotline procedure that we already have in place in our state to address the credible-allegation issue in the 2301 memo. So I asked directors, and we have about 160 LEAs in our state, so I asked the directors to nominate one another to participate on a work group to be able to address the issues and come up with policy ... recommended policy that I could take to the state board of education related to those two issues, the credible-allegation issue and the transfer issue. About 17 directors nominated themselves or someone else to participate with me, and we had several meetings. We all reviewed the guidance together front to back. LauraLee and I went through and made a PowerPoint that hit all of the key issues in the guidance, so we reviewed all of the guidance, and then I had the two questions that this work group needed to address at the end, and in the almost 14 years that I have been at the state board of education, that work group did, I think, the best, the quickest work that I have experienced at the state board. They all agreed to just drop their defensive about the fact that this ... The whole credible-allegation issue was going to be more work for them because we have a statewide hotline-complaint system, so I get allegations all the time.

00:11:18.79 >> Yeah.

00:11:19.57 >> And so we just got to work. We probably spent 3 hours together over two and a half different meeting times, and they came up with a decision tree that I then took to the state board of education as a recommendation for how we would proceed, and they all agreed. They asked great questions. I was just so proud of them. And the state board accepted it, and we are moving forward with putting together a database for tracking the procedure that they outlined and that the state board accepted, so we are ... we're beginning the implementation of that system now.

00:12:11.04 >> Wow, that is fast. That's amazing, and, yeah, what you went through to come to that and bringing in the directors to really formulate your plan makes so much sense since they're going to be the ones involved as well and their buy-in, so that's really smart. Well ...

00:12:31.02 >> It was amazingly smooth. I'm really proud of them.

00:12:34.37 >> Yeah, that's just a testament, I think, to the relationship like we've been talking about that you've built with them and other stakeholders over the years that you can tap them in that way and that it is such a successful activity. Great. Well, I want to just thank you both so much for your time and talking to us about all of the great work that you're doing and the general supervision elements that you already have in place and what you're working on. It's so good to hear from both of you. Thank you so much.

00:13:08.87 >> Thank you, Amy.

00:13:10.03 >> Thanks.

00:13:12.23 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.61 >> Hello, welcome to "A Date with Data" with Differentiated Monitoring and Support, or DMS 2.0 in full swing, and the general supervision Q and A that came out from OSEP back in July of 2023. General supervision is a high priority area for all states. And on this episode, we have joining me Leah Voorhies, who is the Assistant Superintendent of Student Support and also the State Director of Special Education, and also LauraLee Gillespie, who is the Special Education Coordinator of the Utah Program Improvement Planning System, and they are both with the Utah State Board of Education. They are going to be sharing with us the story of their state's general supervision system, how they use it to monitor and support compliance with federal and state requirements in local education agencies across the state of Utah. Welcome, Leah and LauraLee. Thank you so much for being here.

00:01:24.09 >> Thank you.

00:01:25.10 >> Thank you for having us.

00:01:26.52 >> Of course! Can you start out just really briefly introducing yourselves and saying a little bit about your role?

00:01:33.38 >> I can begin. I'm Leah Voorhies. I've been the Assistant Superintendent and State Director for going on 8 years now. Before that, I was the Special Education Coordinator, so like the assistant director in Utah, for 6 years. And I have been working on trying to improve and implement general supervision systems in either LEAs or the state for ... This is my 22nd year.

00:02:08.98 >> Mmm, you've a long history of it.

00:02:13.10 >> And I'm LauraLee Gillespie. I've been at the State 6 years now. My work has been around program improvement and compliance for the most part, the entire time I've been at the State. Previous to the State, I was at the Disability Law Center, which is the protection advocacy for the state of Utah, and representing parents of students with disabilities in special education.

00:02:39.38 >> Great. Well, I guess I want to start out just, if you could, talk about your more data-focused areas of your general supervision system, like the SPP/APR, data on results and processes, LEA determinations, just describe those areas to me and what you're doing in terms of monitoring and compliance.

00:03:00.75 >> I will start this part off. So we focus on data a lot ...

00:03:05.28 >> Mm-hmm.

00:03:05.55 >> ... through lots of different avenues. We focus on the indicator data as outlined in the annual performance report. We also look at state-level data and local education agency-level data as we work with local education agencies to improve outcomes and ensure compliance. Just for clarification, in Utah, charter schools and districts are all public schools, so we just refer to all of it as local education agencies. Some areas that we prioritized in Utah have included post-secondary transition planning. That's our ... It's just been an area of focus that we've needed to really focus in on, really looking closely at the data and trying to improve the outcomes for students with disabilities. We've done a lot with inclusive practices and effective instruction. We're also starting to do some educational benefit reviews and looking at the data of what we're finding from LEAs across 3 years. We're just starting into that. We're in our baby phase of educational benefit, and we just want to look ... We want to educate our local education agencies on the importance of monitoring progress over time as well as developing meaningful and rigorous individualized education programs. So again, looking at indicator data across the board, looking at specific local data as well as state data to try to really get a whole picture. Currently, our LEAs annual performance determinations are based on the 16 indicators, so 1 through 16 as they are placed on the annual performance report.

00:04:51.99 >> Mm-hmm.

00:04:52.21 >> We go through and bring those together and develop a score, and they get a determination, but we also do a results-driven accountability or a risk score as well as that APR determination. And the risk score includes 1 through 14 of the indicators, but we also take into account fiscal data, timely and accurate submissions of data, the scoring on the completeness of the program improvement plan, if they've had corrective actions plans.

00:05:26.08 >> Mm-hmm.

00:05:26.37 >> And part of the reason for that is to help them to ... to help us to see how we can best support them. So the higher the risk level is, the more support the LEA is offered through the state board of education. I can jump into a little bit further about our monitoring, but I want to know are there any questions at this ...

00:05:49.08 >> Yeah, I was just ... I was going to ask that, yeah.

00:05:51.34 >> Yeah.

00:05:51.46 >> In terms of the risk score and how that feeds into the monitoring and how you decide which districts would ... or LEAs will get monitored and when?

00:06:01.73 >> Yeah, so we've gone through some overhaul in the last 3 years which has been a really great change for us. We switched to ... I want to call it a 6-year cycle, but it's not a cycle as in, I'm going to come to your LEA, and you can just say, "Oh, I'm off the hook now for 6 years."

00:06:21.70 >> Mm-hmm.

00:06:21.95 >> The way we look at it is, we tell the local education agencies that we are going to come at least once every 6 years. We may come more. We may come less, and this is where the risk score comes in. So the risk determination is considered as well as several other factors. So the higher the risk, the more likely we are going to be coming to them.

00:06:44.42 >> Mm-hmm.

00:06:44.78 >> But we also look at other pieces as well. We are really focusing in on our targeted support and improvement schools, or our TSI schools. We look at how much dispute resolution they've had.

00:06:57.33 >> Mm-hmm.

00:06:57.88 >> We have a hotline here at the State Board of Education, so we pull that into the determination of who we're going to be monitoring during the year. If there's fiscal concerns, we take that into consideration, and that's sometimes a specific population. So our youth in custody, our youth in care student populations are always a focused part of ...

00:07:22.77 >> Hmm.

00:07:22.89 >> ... who we do full monitoring with, but it may increase the likelihood that we're going to come a little sooner rather than later.

00:07:30.25 >> Hmm.

00:07:30.35 >> So again, it's a 6-year cycle, but the cycle is, you can be moved up quicker depending ...

00:07:37.06 >> Mm-hmm.

00:07:37.36 >> ... on the risks and needs that we see based on your data and other factors.

00:07:43.71 >> A very holistic approach, it sounds like ...

00:07:46.39 >> Mm-hmm.

00:07:46.60 >> ... looking at comprehensively all factors to make those types of decisions.

00:07:52.58 >> Absolutely, and that's our full monitoring. And, of course, we do additional monitoring for fiscal ...

00:07:58.57 >> Mm-hmm.

00:07:58.79 >> ... Indicators 4, 9, 10, 11, 12, 13, and our significant cognitive disabilities are 1 percent. And those aren't based as much on risk. They're based more on data and information that we have on file regarding those specific indicators, determines who we look at.

00:08:18.61 >> Great. And through all these different areas that you're focusing on, can you talk about how stakeholders are involved in either making some of the decisions around what these different processes may look like, reviewing results, giving feedback? How do you bring them in?

00:08:38.01 >> So this is Leah, and stakeholder feedback is important to us because it's required.

00:08:46.44 >> Mm-hmm.

00:08:46.65 >> It's also important to us because we're a small enough state that we actually know all of our major stakeholders. We participate with them in all sorts of activities, not just in gathering feedback related to the activities that we do.

00:09:06.44 >> Mm-hmm.

00:09:07.17 >> So we have established as an entire agency, not just in special education but as an entire education agency, feedback gathering standard operating procedures. So ...

00:09:21.53 >> Hmm.

00:09:21.66 >> ... For all of the decisions that our elected State Board of Education makes, we have a process to gather feedback. And then in special education, we ... Because we know all of the frequent players when it comes to students with disabilities and individuals with disabilities, we interact with them frequently. So I meet monthly with the director of our parent training and information center.

00:09:55.51 >> Mm-hmm.

00:09:56.76 >> My team members meet monthly with the PTA, with the board of the parent training and information system. We meet regularly with our Disability Law Center. We have a legislative coalition for people with disabilities that we meet with. I serve as the chair of a statutorily required committee on policy, state policy related to students and individuals with disabilities. All the state agencies in this state are required by statute to be ... participate in that committee, as well as all the nonprofit organizations that serve individuals with disabilities that receive any state funding.

00:10:41.83 >> Hmm.

00:10:42.30 >> And then we have a robust focus group and follow-up survey system that we use. So our first strategy is that we go to our stakeholders. We don't ...

00:10:56.50 >> Mm-hmm.

00:10:57.23 >> ... require that they come to us. And so we go to their meetings. We participate in their meetings as regular members, even voting members most of the time, and then we hold focus groups with them when there is something specific that we need them to comment on, and then we have a follow-up survey. And the focus group, the follow-up survey process is an agency process. That's not just special ed.

00:11:27.92 >> Yeah, I like that idea a lot. I think states do seem to see more success sometimes going to the stakeholders rather than adding something else, another meeting or time on them, but hitting them when they're already gathering or meeting for other purposes.

00:11:47.06 >> And if I tried to roll something out that they hadn't had ... that they hadn't provided feedback for, it wouldn't go very well.

00:11:58.15 >> You'd hear about it?

00:11:59.59 >> We have created ... They're my friends. They're my team members, friends. We have ... Luckily, I've been the State Director long enough, and my team has been ... LauraLee has been with us for long enough that we have a longstanding team. We have a cohesive team. They know us. They know us personally. We know them personally. It's not just that we respect each other. It's that we're friends, and it would damage our friendships, and that is not okay with us or with them ...

00:12:36.08 >> Mm-hmm.

00:12:36.70 >> ... for that matter.

00:12:37.75 >> Yeah, so building those relationships ...

00:12:39.94 >> Yeah.

00:12:40.64 >> ... is very key. What are some of the aspects of your general supervision work that you're especially proud of, that you really want to highlight?

00:12:51.33 >> There's a lot of things. It's been a ... As Leah said, the stability has led to some opportunities in terms of general supervision that have worked well. In no particular order, when this question was asked, I started to think about these things. One thing that we do in terms of monitoring that I'm really proud of is, we've really worked hard to, as one of our colleagues says, "Practice supports compliance, and compliance supports practice." And we've really tried to ensure that that is evidenced by the work that we do at the State. So in other words, when we go out and do monitoring, our specialists, our content specialists in effective instruction and inclusion and behavior, they come right along with us on those monitoring visits. They conduct the interviews, and what I see almost every time we go on a monitoring visit is that we start out with this very nervous feeling ...

00:13:57.23 >> Mm-hmm.

00:13:58.42 >> ... amongst the staff, and that doesn't always go away. But a lot of staff build relationships with our specialists during those monitoring visits and actually follow up with them and have conversations with them about their concerns. And so that ... Again, it's that building a relationship, and our monitoring is all on-site, and that's very ...

00:14:21.90 >> Okay.

00:14:22.07 >> ... intentional because we want ... We really want to build those relationships with those teams. Just as an example, our assessment specialist, she had a very in-depth conversation with one of our rural school districts, and they've continued to communicate with each other based on the monitoring visit.

00:14:42.35 >> Hmm.

00:14:42.47 >> And Indicator 13 is another really great example. We have this amazing coach for Indicator 13. She does just an incredible job, and as they go through and do the file reviews, she's right there. And they're doing the file reviews and they're checking compliance, but she is right there giving them tips and ...

00:15:04.20 >> Mmm.

00:15:04.38 >> ... things that they can use as they move forward. And again, that, to me, is powerful in terms of trying to really improve outcomes for students and think about it through a different lens. We've been doing a big focus on inclusion here ...

00:15:24.33 >> Mm-hmm.

00:15:24.55 >> ... in the state of Utah, and I think that's starting to meld into some of our monitoring and our general supervision. A lot of the technical assistance documents, we can immediately pull those as we go out and are monitoring or helping an at-risk school to ... and again connecting them to folks at the state who are specialists in these fields. Another piece, again I'm going to back to educational benefit. We're newbies to this. There's other states out there that have tried it, but our educational benefit specialist recently was with a school and went through that process, and the power that was instilled in the actual LEA, not by the state, not us telling you what to do, but the LEA, the staff there realizing and recognizing what they needed to do to really improve progress for students with disabilities was ... It's just phenomenal. And the last thing, I think, is the internal thing that we do as a state. We, for the last 4 years, have done general supervision sessions. So this isn't so much so much for the LEAs on the outside but really for our staff on the inside, and this was ... Again, I have to give Leah credit for because she had this wonderful idea that we need to ensure that all of our staff understand general supervision ...

00:16:50.03 >> Mm-hmm.

00:16:50.69 >> ... that golden requirement and why we exist as a state agency. And it's led to ... We just meet once a quarter, and we train out on different components on the general supervision pieces. We talk about rules. 2301 has been a big focus this year [Indistinct] and will be next year again ...

00:17:13.24 >> Mm-hmm.

00:17:13.47 >> ... because it's the new thing that is related to general supervision. So I kind of belled on, but I have a lot of things that I'm very proud of that have been happening. Leah, I don't know if you have any additional thoughts.

00:17:28.68 >> No, that's a good overview of things that we are proud of. I think where you started, LauraLee, with the fact that we have a team that has been together for a long time and that, as a result of that, we've been able to be very intentional about our continuous improvement process.

00:17:56.80 >> Mm-hmm.

00:17:57.03 >> And we've been able to be intentional with our LEAs and with our stakeholders about our continuous improvement process and support each other, and with the IETs that are training and preparing new special education providers. So that's ... I think that's definitely one of the things that we're proud of is, because we have been around a while, we have established relationships that are making it easier for us to improve our processes, and hopefully that leads to improved outcomes for students.

00:18:33.94 >> Yes. Well, it sounds like you have a lot in place to be proud of, for sure, and that sustainability seems like a major part of that because of building those relationships and maintaining those relationships and that trust, so kudos.

00:18:49.69 >> I think Leah brings up a really good point about the institutes of higher education and efforts to really prepare teachers. It's looking at it again from a whole picture, right, of ... It's not just, what can we do with the LEAs? It's, what can we do with those who are coming into this field to improve that as well?

00:19:09.41 >> Thank you so much, Leah and LauraLee, for sharing with us all of the great things you have going on in Utah around the data elements of your general supervision system. Please make sure to listen to our next episode of "A Date with Data," when we will be hearing more about your general supervision system and all the wonderful things you have going on.

00:19:36.65 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Resources

Response Rate Representativeness and Nonresponse Bias

Nonresponse Bias Analysis Application

00:00:07.34 >> Hey, it's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.70 >> Welcome to "A Date With Data." On this episode, I am joined by Heather Dunphy, who is the lead education programs specialist from the Arizona Department of Education, and we also have with us one of my wonderful colleagues from IDC, Tamara Nimkoff. Heather is going to be talking to us about how Arizona has been conducting their nonresponse bias analysis that all states are required to complete for SPP/APR Indicators 8 and 14, and Tamara is also going to be here to highlight an IDC tool and talk about some other support that IDC can provide states related to the nonresponse bias analysis so welcome to both of you.

00:01:08.06 >> Thank you.

00:01:09.19 >> Thank you.

00:01:10.50 >> So first off, I was hoping if each of you could just introduce yourselves briefly, say a little bit about your role and what you do. Heather, do you want to go first?

00:01:20.13 >> Sure, my name is Heather Dunphy, and I'm a lead education programs specialist at the Department of Education in Arizona. I work a lot with significant disproportionality and LEA determinations, and I also coordinate most of our federal submissions, including the state performance plan annual performance report, and I'm really happy to be here today.

00:01:42.19 >> Great, happy to have you. Tamara, do you want to talk a little about what you do on IDC?

00:01:47.26 >> Sure, thanks, Amy. So I've been with IDC as a state liaison for mostly a [Indistinct] specialist for several years now. My work has been really focused around the analysis and use of data, previously on the state systemic improvement plans, on using data on structured data meeting and over the recent years more focused on supporting states around their data collections or sampling as well as in the areas of representativeness and nonresponse bias, on the topic of today's chat.

00:02:26.75 >> Great, thank you both. So, Heather, can you start us off by kind of walking us through Arizona's data-quality journey related to nonresponse bias analysis? What does that look like?

00:02:39.97 >> Sure, so I began with the agency about 2 1/2 years ago. The FFY2020 SPP/APR was the first federal document that I was responsible for coordinating, and that was the first year that the words nonresponse bias appeared in the APR, and the question in the APR, it asked, describe the analysis of the response rate including any nonresponse bias that was identified. So I was familiar with how to analyze response rates, but I was unclear what exactly nonresponse bias was and certainly how we were going to analyze it. So in 2021 we did our best to analyze the nonresponse bias to the extent that we could, so for Indicator 8, for example, parent involvement survey, what we did was, we divided our survey window into three periods: the beginning period, the middle and the end, and the idea was that the responses might differ from people who answered this survey early compared to those who answered the survey late, and then we examined those responses that came in from parents at the end of the data-collection period as a proxy for nonresponders. And then we then compared those responders to the ones that came in during the beginning and the middle of the data collection period, so this method gave us some insight into whether or not the results might be biased. And then the other strategy we used at that time for Indicator 8, we were looking at our responses by subgroup and to see if they were representative in respect to certain demographic areas such as race and ethnicity, and then we looked at the rate of agreeableness with Indicator 8 by race and ethnicity, and we were just kind of visually trying to see if there was any nonresponse bias. If we received more survey responses from one particular race, ethnicity, and their level of agreement was different than the others, then there might be nonresponse bias. So this provided a good estimate for measuring nonresponse bias at that time. Those are kind of the tools that we had at that time.

00:04:58.87 >> Gotcha, okay, what about for 14? Were you doing something similar??

00:05:02.10 >> Yes, we were doing the same for 14 that we were doing for, or similar, to Indicator 8, yes.

00:05:09.91 >> Great, so you had kind of your start and had a sense of what kind of makes sense. And then what led you to engage with IDC in terms of supporting you around the nonresponse bias analysis? I know you attended the Hands-On Learning Academy that IDC hosted on the nonresponse bias analysis tool that IDC had developed so tell me kind of what made you kind of shift from what you had been doing initially to wanting to do something different, something more.

00:05:40.62 >> Mm-hmm, right. Well, yes, in the spring of 2023, it was last spring, I had heard about a new tool for analyzing nonresponse bias that was in the testing stage, and they were looking for states to try it out.

00:05:56.98 >> Mm-hmm.

00:05:57.72 >> I had heard that the tool was built to assist states in addressing the requirements related to response rates, representativeness and nonresponse bias and to kind of ease that burden of analyzing survey data for Indicators 8 and 14, and anything that's going to ease the burden of any work, I'm all for it.

00:06:18.89 >> Mm-hmm.

00:06:19.85 >> So myself and one of our Indicator 14 specialists in April went to Rockville, Maryland, for 2 full days of learning about the NRBA App.

00:06:31.23 >> Great, and you said you did attend that?

00:06:35.21 >> Yes, I did.

00:06:35.90 >> The OLA. Okay.

00:06:36.71 >> Mm-hmm.

00:06:37.15 >> Do you want to talk a little bit about your experience at the OLA, what you learned?

00:06:43.00 >> Oh, absolutely, yeah, at the workshop it was Tamara and Ben. They walked us through the various ways to use the app. There are several tests that we can run. They showed us how to set up our data set in the appropriate columns. It needs to be set up in a certain way, and as soon as you have your data set up in a certain way, really the app does most of the work, and I think Tamara can talk more about all the things that this app does, but it really does take the burden off of the user and puts it onto the computer to do the calculations for you.

00:07:20.76 >> Yeah, that's always a good thing in lots of ways. We can hope there's not the user error that we might experience, and it's just kind of pushing a button.

00:07:29.59 >> Right, and the nice thing about the workshop was, it was still a little bit in that testing phase, so we had a small group of representatives from several states, and we could try it out, and we saw that some things were not quite right, and then that gave them time on the developer side to work out the kinks before it went live to all of the states.

00:07:50.98 >> Great, that was, yeah, a good kind of dual opportunity there for IDC to be able to have some testers, too.

00:07:58.39 >> Mm-hmm.

00:07:59.61 >> So, Tamara, can you talk more about the NRBA tool, how it works, why we developed it, how states can use it?

00:08:08.48 >> Yeah, absolutely, and building off of what Heather has shared, really emphasizing that the development of this resource has really been directly informed by input from our colleagues and in the state agencies. We were at the beta version stage in that spring 2023 that Heather mentioned attending. We had other states, representatives from Georgia, Indiana, North Carolina and West Virginia there, and we also had earlier input on an alpha version from our friends in Montana that gave really valuable feedback to inform the development. The impetus of the tool was really ... It's really aligned with our direct support of states' capacity to meet those SPP/APR data quality requirements for Indicators 8 and 14, as Heather was sharing about the work that they were doing prior to engaging with this particular tool, and it was triggered really by those requirements of the 2020 package where it caused us to think about at that time what tools were already in the field and where the gap might be. We felt that the field could really benefit from a tool that was both powerful but also flexible and user-friendly to the extent possible, so we wanted something that would allow a user to choose among many different ways of analyzing their survey data using those best practices, but that gave them some flexibility to have as much guidance along the way as was needed. So it was also informed by some of the kind of common issues that we at IDC had observed over years of supporting states in writing their SPP/APR responses to those prompts, and we knew that we wanted something that could both sort of support states in getting that conceptual understanding of the differences between data representativeness and nonresponse bias, which Heather alluded to, as well as gave them options for digging into their data in deeper ways beyond the submission of an APR report each year if they wanted to. So a little bit about the tool itself just to give people a really high level: It's a browser-based application, online and application, and the first time that people use it, they'll install a free statistical program along with the package itself. It's the program R, which is an open-source program that's widely used across many fields, including education. A couple of things that were really important to us were to make it really flexible but secure, so users access the app within their preferred Web browser like Google Chrome or whatever while the statistical program runs the computations in the background.

00:11:44.73 >> Mm-hmm.

00:11:45.35 >> When users upload in order to use it, their data into the app for a session, it's done via a Web browsers, so no data are actually passing across the Web, so their data remains secure within their local computer.

00:12:04.58 >> Yeah, that's important.

00:12:06.17 >> Absolutely, absolutely, it's kind of built on, the application, it is built on a couple of kind of stages. It guides the user through setting up the session, which is importing their data set and then indicating how they collected those data, telling the application about their data. For example, was it from an attempted census? Or was it from a sample? Then they can choose from a whole series of, excuse me, analysis options, their questions about response rate, representativeness and nonresponse bias. So it's a tool that is not just focused on nonresponse bias but allows the user to look at all of those areas guided, for example, what are our response rates? Do they differ across subgroups? Are some subgroups in our population overrepresented or underrepresented in our data? Is looking at data representative? But then also how do our survey outcomes differ across subgroups? And understanding how those survey outcomes vary across subgroups combined with the information about representativeness of their subgroup is what informs the user about the presence of nonresponse bias in their data.

00:13:36.13 >> Mm-hmm.

00:13:37.16 >> And the tool is also powerful enough that it gives the user options for looking at, can we use some statistical adjustments to reduce nonresponse bias of the data?

00:13:47.94 >> Hmm.

00:13:48.24 >> So not a requirement of the APR but an added kind of best practice of looking at one way of assessing if there's nonresponse bias and seeing how it might be adjusted is by using weighting adjustment.

00:14:05.42 >> Mm-hmm.

00:14:06.14 >> So that's an option in the tool. There are many kind of analysis options that if a user wanted to dig more deeply into their data, they can do so.

00:14:17.62 >> So, Tamara, would the tool actually weight the data for you or [Indistinct]?

00:14:22.64 >> That's right. That's right. All of the analyses, whether it's calculating a response rate or whether it's looking at the proportional difference between representation or whether it is providing a comparison of weighted and unweighted data, all of that is done through these preprogrammed analyses that are part of the application. They're running in the background, and the user is choosing what variables to look at as well as other kind of parameters of the analysis that may need to be decided depending upon the specific statistical task that's being done.

00:15:13.13 >> Wow, that's ... And just having that all in one package, all of those pieces because I think so much of the confusion that we heard from states, especially when the nonresponse bias analysis requirement was added, was not understanding the difference between them, how they are connected, how they work together potentially, and so just having it all together like that I think is so powerful for understanding the different requirements.

00:15:41.52 >> That's right. Yeah, that was really a goal, and one of the things Heather mentioned, the input that was provided at the OLA, which, again, I'll just say was so valuable for our development process, one of the things that is also really a part of the application are the supporting resources around it, so having resources that provide detailed instructions about not just how to install the app but how to set up the data set for really honestly whether you're using the application or not, what elements of the data set are valuable to have for the particular analyses that need to be done for them as well as kind of a pretty comprehensive reference guide that gets to the conceptual pieces of along with using the app, what it means. How do you interpret the analyses?

00:16:54.75 >> Yeah.

00:16:54.84 >> So all of the pieces we want to be a part of the resources available and we're continuing to think about and get input from folks about what resources might be useful moving forward.

00:17:13.25 >> Yeah, and this is something that you don't have to have any type of statistical background to use. What would you say ... A data manager could pick this up and do it with the resources ...

00:17:25.51 >> Yeah, I ...

00:17:25.79 >> ... and support that goes along.

00:17:28.33 >> ... Yeah, absolutely, and Heather can certainly speak to this as well from our, the great engagement that we've had in collaboration around them using this tool with their data is that we really do encourage users to leverage IDC's technical assistance to make the most of the application, but the support is really intended to be flexible. There may be folks who are very experienced in statistics who might choose to use the application independently just using the guide for reference. Others may really benefit from engage one-on-one with an IDETA specialist like myself to work with the data collaboratively to get input on which analyses might be most useful to them or to discuss together, how the results might be interpreted. Folks might want to kind of gain proficiency with the tools themselves with that range of IDC support, and it really is ... The support around the tool is really quite flexible as well.

00:18:43.18 >> So, Heather, tell us about your experience using the tool and what that's been like and how maybe that's changed your analysis, your interpretation, your results in terms of the nonresponse bias and representativeness as well.

00:19:00.24 >> Sure, sure, so like I mentioned at the workshop, Tamara and Ben, they walked us through various ways to use the app, and just like anything we learn that is new, I felt a little outside of my comfort zone at the workshop but with their leadership, the process wasn't too stressful. It seemed fairly straightforward, but then when I returned to Arizona, I tried using the tool on my own. I found it far more difficult than I anticipated. I felt clumsy. I was making mistakes. I tried to follow the written directions I had gotten at the workshop, but I was stumbling with the steps, so I reached out to Tamara. I felt a little embarrassed because I needed additional support after spending 2 fulls days with me but you know what? She made me feel totally comfortable. We went through some of the exercises together. She helped me understand how to interpret those results. She made sure that my data set was organized correctly, and then after that the process was fairly simple. Just like Tamara said, the RStudio app will run all of the calculations, so it can do some of the basic calculations like calculating the response rates by subgroups and comparing subgroup percentages in respondent data to data from respondents and the nonrespondents, but what I found really helpful is learning, do we have that nonresponse bias? And so, for example, in Arizona the Indicator 8 data we have is about 92 percent of agreeableness, but we're only receiving 14 percent of responses, so that's a lot of unknown. That's 86 percent that we don't really know how they're going to respond, and so the tool was really helpful. We found that if we were to get everyone to respond, which we only got 14 percent, but if we were to extrapolate that and get everyone to respond in respect to race and ethnicity, it was extremely close to that 92 percent that we calculated. So that gave us ...

00:21:08.09 >> Good.

00:21:08.25 >> That gave us a lot of confidence that we didn't have nonresponse bias in respect to race and ethnicity for Indicator 8, and we also looked at Indicator 14, and we received quite a few responses. We have about a 74 percent response rate, but you don't still don't know if there's nonresponse going on even though we have quite a few respondents, so we ran the test there, and what we found is, we're getting about 80 percent of our responses from graduates, okay? So we get quite a few of the graduates, but we're getting fewer responses from dropouts. It's harder to get responses from the dropouts, and so we wanted to know if there's some nonresponse bias going on. And the areas of engagement differ from graduates and dropouts, and especially when we look at, for example, who's going into higher ed, what we found is that for the percentage of youths with IUPs that go into higher ed, it's about 19 percent in Arizona. But we were wondering, if we received responses from everyone, would it still be 19 percent? So that's where the tool really came in, in handy, so in a perfect world, if we had gotten all of the responses from graduates and all the responses from dropouts, the app showed us that instead of 19 percent, it would be lower. It would be about 17 percent.

00:22:31.25 >> Mm-hmm.

00:22:31.88 >> And so my next step in this process would be to learn about, is that significant? What's the level of significance, that 2 percent difference? And that's something that I think the tool can help with. I would like to learn more about that in the future.

00:22:47.34 >> Mm-hmm.

00:22:48.91 >> So I'm still kind of at the beginning stage of learning about the tool, but from what I've seen it is really, really neat and saves a lot of work.

00:22:57.37 >> Yeah, did you use it for the SPP/APR you're working on that's due in February?

00:23:02.25 >> Yes.

00:23:02.46 >> Great.

00:23:02.76 >> Yes, we did, mm-hmm.

00:23:04.11 >> That's very exciting. Well, it sounds like, yeah, you've gotten a lot out of the tool, and there's a lot it can do and a lot more still to even explore with it, so kudos to you all.

00:23:17.68 >> Yeah, thank you. I will add that any time you're working with something complicated, you try not to make it so complicated. You try to simplify it, so one thing that did help me to understand the tool better is, I made a smaller data set of fake data. So instead of looking at 10,000 responses, I just made a false data set of 100 responses to help me just understand the tool better. And so in respect to race, ethnicity, I just made 50 were Hispanic, and 50 were white, and I ran different scenarios to see how the tool worked and what if more Hispanics answered? What if more ...

00:23:57.06 >> Yeah, how would that change?

00:23:57.95 >> ... white people answered? What if the Hispanic level of agreements was very high, and the white level of agreement was really low? How does that change each of these calculations. So I would suggest to anyone learning about this tool to try that, to create a smaller data set and just look at the different statistical calculations that the app provides and what states might find valuable.

00:24:19.09 >> That's a great tip. Do you have anything else for states that are just interested in possibly using this or have starting using it that might be helpful for them?

00:24:28.87 >> Sure, I would say in addition to trying a small data set, to reach out to IDC. Like I said, I was kind of unsure, kind of feeling clumsy about it, and they are just so helpful to make sure that your data set is set up correctly. If it's not, the app is not going to work at all, and so that's the first step. And then when you finally get the results, you might need some help interpreting them, and that's where IDC can really come in handy as well.

00:24:54.46 >> Great, so, Tamara, for states that are interested in learning more about this and getting their hands on it, can you tell us how they can get the tool and maybe a little bit more about the support that IDC is able to provide?

00:25:09.45 >> Yeah, absolutely, well, the app itself, information about it, if they want to kind of self-explore is available on the IDC website along with those supporting documents. Also on the website in various places are various kind of presentations that we've done either at the SPP/APR summit, for example, that speak to both the topic as well as a bit about the application. If a state user doesn't want to kind of engage in it, then contact your IDC state liaison, and they will be able to connect you with a TA specialist. In terms of what's next, we are continually improving and looking for those opportunities to provide TA, really, that help inform the hearing from the field on what other kind of supporting resources would be useful. It's been immensely helpful to collaborate and rewarding to collaborate with Heather in Arizona and, for example, mentioning the smaller fake data set, we have small data sets that we have developed that are, with fake data, that are for Indicators 8 and for Indicators 14 that we use for demonstrations with the app, and we certainly could make those available to folks so that they could then ...

00:26:43.24 >> They ran themselves.

00:26:44.27 >> ... use those, absolutely, so we're constantly learning from each other in this process, so that's great, and as she mentioned and I mentioned, really our support is very flexible and geared towards what's going to be the most useful and meaningful for the state user in whatever phase they are in, in this process.

00:27:09.43 >> Great, and we'll put links to the resource application itself as well as some of the other presentations that might be helpful in the notes for the episode so folks can easily get to them.

00:27:21.77 >> Great.

00:27:22.56 >> Great, well, thank you both so much. I know that I learned a lot more about the app than I knew before, and it seems like something that's so useful and helpful and so appreciative that you came on and have shared your story so other states can hear about this, too.

00:27:39.92 >> Oh, you're welcome. Thank you for having us.

00:27:41.84 >> Yeah, thank you so much.

00:27:42.96 >> Of course.

00:27:44.71 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

00:00:07.34 >> Hey. It's Amy, and I am so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who just like you are dealing with IDEA data every day.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.87 >> Hello. Welcome to "A Date with Data." On this episode, I am joined by two data managers. We have Amy Patterson from the Kentucky Department of Education and Alisa Fewkes from the Idaho State Department of Education. Both Amy and Alisa have been in their role for a number of years, and they're going to be reflecting on their experiences and also sharing some lessons that they've learned over time and insights that they have for newer state staff. Can each of you tell me how long you've been your state's data manager and maybe also a little bit about your responsibility since we know that a lot of different tasks and initiatives fall under being a data manager, and that can really vary a lot from state to state, what you're responsible for? So love to hear what you do in each of your states.

00:01:14.39 >> Sure. So my name is Amy Patterson, and I am the Part-B Data Manager in Kentucky, and I've been in this role for about 8 1/2 years. Some of my responsibilities include pulling data from our statewide student information system as well as getting some data from the districts directly. And then I review the data, format the data, and then I give it to our effects coordinator, and she's the one that actually submits the data. I also do a lot of training to LEAs on how to enter data and how to review their data and so that we get reporting ... quality data reported back to us. And I do the data for the SPP/APR, but I as much as not responsible for writing the SPP/APR or the SSIP.

00:02:05.67 >> Got you. Okay. Thanks, Amy. How about you, Alisa?

00:02:08.78 >> Hi. I'm Alisa Fewkes. I am the Data and Reporting Coordinator for Idaho, and I have a lot of similarities to Amy. I've been in my position for right about 8-ish years. I do work directly with LEAs to provide support on their data quality for especially child count and then those bigger reporting areas, program exit, all the way through. So I can be contacted by our LEAs to kind of go down those rabbit holes of data and reporting coding. I also provide support through doing trainings, working with teams on data quality issues, figuring out, can we ... more data and reporting plans that they can consider, those root causes to go through and look at all of their data and information. We always provide a regional training every year that I'm part of, which goes around like five different locations around the state and go through root causes analysis. And then I am also the lead for SPP/APR, both on data, writing, submission. Thankfully, as Amy said, don't do the SSIP ...

00:03:41.57 >> Mm-hmm.

00:03:41.78 >> ... so very glad about that. We have other folks working on that information. And then I work directly with our IT team for pulling data for our EDFacts reporting, so do a lot of work with them. If we're seeing any reporting errors as we go through the submission, I troubleshoot with them trying to figure out what's going on.

00:04:05.47 >> Mm-hmm.

00:04:06.11 >> And so a lot of direct interaction, and most of the reason for that is actually because I've pushed for it ...

00:04:14.62 >> Hmm.

00:04:15.32 >> ... for my position. I really want to be more hands-on as opposed to just having my data come in and not knowing what's going on with it. So I really appreciate that team members have been willing to let me be involved and that I have a good relationship with our IT team who's involved in all of that data collection and processes.

00:04:40.51 >> Hmm. Thank you both, and what you were just talking about, Alisa, I think kind of folds into the first question I had, which is, how has your role evolved over time? And it sounds like maybe things have changed somewhat. I'm sure they have for both of you both in terms of what's going on in your state and the context and other staff that are there as well as SPP/APR changes, EDFacts changes. So, Alisa, do you want to kick us off just saying a little bit how things have changed since you started about 8 years ago and until now?

00:05:13.99 >> Yeah, sure. So when I first came in, there was a lot of processes being run without interaction of the Part-B Data Manager, which, again, I just wanted to be a lot more hands-on. I like knowing.

00:05:31.76 >> Mm-hmm ...

00:05:32.26 >> So ...

00:05:33.23 >> ... what's going on.

00:05:34.60 >> Yeah, so I really work to build up those relationships and team interactions across our different divisions, so whether it was with IT or assessment, those different folks to make sure that when data processes were being considered for changes that they were really, "Oh. We need to bring Alisa in on this conversation." So it wasn't like that special ed was outside of that conversation. It was automatically, "Oh. We need to make sure that we're including those folks, and I'm right in there with them." And that was a big change when I came in. That was very much a, "Oh. Oh, yeah. We didn't talk to special ed. maybe we should do that."

00:06:31.80 >> Mm-hmm.

00:06:32.67 >> It's just a regular part of the process, which I think is great. And again, I just like knowing, so to be in the know, you have to be part of those problematizations ...

00:06:46.79 >> At the table.

00:06:47.62 >> ... yeah, and at the table. The other thing I've seen change over time, I see kind of a pendulum swing back and forth, and sometimes it depends on the time of the year of how much direct support I'm providing to LEA teams. Sometimes it really is more I'm on SPP/APR. I am not having those one-on-one conversations with teams. And sometimes it's really majority of my day is taken up providing feedback on calls to LEA teams.

00:07:29.80 >> Hmm.

00:07:30.59 >> So it's kind of a swing back and forth, but a lot of my work also now is around the public reporting, making sure that that information is out in more understandable ways, which sometimes can take a lot more time.

00:07:48.90 >> Yeah, absolutely. Amy, what about you? How have things changed for you?

00:07:53.78 >> A lot of what Alisa said is true for me, as well. I do spend more time now on public reporting and communicating data and data visualizations. And I'm working with a contractor for dashboards, which, that evolved probably 3 or 4 years ago. And a lot of my time has been spent on that. When I first came in, my role was basically just to collect data and report it, and I was a data person but not a special ed person.

00:08:24.87 >> Mmm.

00:08:25.32 >> So the more I learned and because data is in everything special ed, I have become more involved in sort of programmatic discussions ...

00:08:35.89 >> Mm-hmm.

00:08:36.12 >> ... behavior discussions. I am sort of the face of our special ed office because I do most of the trainings, although they're around our data system. People know they can reach out to me, so I'm doing a lot of fielding calls and questions. And also, as I've become more familiar with our system and with special education, I've made some changes to sort of update how we do our data collection and reporting. And we're constantly sort of revising the data collection process and trying to document the data cleanup so that we're consistent and we have the same data. We have better data every year based on what we're getting from the LEAs and reporting back.

00:09:27.96 >> Sounds like both of you have really kind of taken this position and made it your own and really run with it and tried to make improvements and expand like both going more into the public reporting, for example. So that's really exciting and something for those who are newer in the data manager role to kind of think about, too, as you're in your role for more time. And what are other areas that you can get into and improve? Talking about new data managers and new staff, Amy, what advice do you have for those who are just getting off the ground as a data manager?

00:10:07.24 >> First, I would say give yourself a little grace. Don't panic because there is a lot to this, and it's ... It can be a little intimidating, knowing you have federal due dates and all this data coming in. So do the best you can, and reach out to other data managers [Indistinct] ...

00:10:31.42 >> Mm-hmm.

00:10:32.58 >> ... or any one you feel comfortable talking to, IDC, if you have questions. Don't hesitate to ask question. The second thing I would say is to document everything, the data process protocols. I'm always talking about the data process protocols because they kind of saved me. We had a data manager in our state that was here for a long time, and when he left, we were kind of lost. And so it's been very helpful. He came back and helped us document these protocols, and then I've kept them going over the last few years, and it was really helpful. And then, third, I would say relationships are key. And Alisa sort of alluded to that earlier. My relationships that I try to build, I have really good relationships with our LEAs, and I'm building relationships with other offices within our department, our tech team and assessment because those relationships really do go a long way in helping you get your job done and ensuring that you have the best data possible reported.

00:11:47.52 >> Great advice. Alisa, what advise do you have?

00:11:52.24 >> My first one was very much going to echo Amy. Be kind to yourself. It's probably going to take you about 3 years before you're going to feel stronger in your position and that you're going to be able to make changes to your processes. Yes, probably after that first year you're going to make tweaks to what you are doing and going, "Oh, this was a headache. I need to address this." But then after that 3 years, you're going to likely be able to a lot better manage your calendar, getting that calendar set up so that you're working with special ed teams and IT teams, whoever you need to work with so you can really chunk out your processes so they're more manageable because there can be a lot of stuff that mounds up on you all at once if you don't do that. So develop a calendar. Figure out what works with your team, whether you do a lot of EDFacts reporting activities, public reporting activities, whether you're on more of those program monitoring or SPP/APR. Figure out so that you don't have too much on your plate all at once because inevitably something else is going to pop up during that same time frame, and it's going to make it stressful for you. So be kind to yourself. Also, work on developing your team's ability to look at data because if you're the only person who's looking at the data and the only access point, which, to a great degree I am, don't do that to yourself. Make it so that your team knows where the information is, how to use that information, and that may take a lot of time up front, developing that, and may take more time at different points in the year. But that's going to save you time and energy throughout the year if your team members know how to access that and feel confident. And looking that information, I think, is, I don't know, a great dream to have.

00:14:26.86 >> And I think that ties into what Amy was saying, too, about documenting everything, so making sure everything, all your processes, procedures, are documented, well documented, and that others on your team are aware of it, have seen it, have looked at it, understand it, too, so kind of building the internal capacity piece.

00:14:46.95 >> And that also builds sustainability ...

00:14:49.78 >> Mm-hmm.

00:14:50.01 >> ... within the team of anybody at any point leaves.

00:14:54.33 >> Right.

00:14:55.17 >> Then, you have those backups. There was something else I was going to say. Oh, yes. Around documenting those processes, there's lots of activities that you're going to be only doing once a year.

00:15:08.15 >> Mm-hmm.

00:15:08.95 >> So having those processes documents and so you can make changes, updates throughout and take the opportunity to make those updates when they're happening because you may forget about it later on, and you don't want to have to go back and do a lot of work on a lot of different files later on.

00:15:32.76 >> Yeah.

00:15:32.91 >> But, yes, document your processes because one year later, you've had a lot of things you've done.

00:15:39.71 >> Mm-hmm. Yep. Yeah. Having ... You're kind of going through and having your processes open and available kind of while you're going through it and updating it along the way, too, as you as you make changes. Like you said, instead of waiting until after, when you feel like maybe I have a little more time but then you're going to forget things. So that makes sense. So, Alisa, I'm really curious to hear, what is your favorite part about being a data manager?

00:16:07.78 >> Favorite part about being a data manager is really that collaboration that you can have across team, those relationships you can build, those opportunities to ... I'm a person who really likes troubleshooting stuff.

00:16:24.78 >> Mm-hmm.

00:16:25.86 >> So going through working with others to make the process better, whether it's more efficient or better quality or whichever, that collaboration and bouncing ideas off of other folks, that's one of my favorite things. The other is when I actually get to do more of that direct support to LEAs, and they have an ah-ha moment.

00:16:52.88 >> Mm-hmm.

00:16:53.39 >> "Oh. You mean I don't have to do all of these things, and I can get my data to look better, and it's going to be easier for me?" That's another one of my favorite things.

00:17:05.26 >> Mm-hmm. Yeah. Amy, what is your favorite thing or things about the job?

00:17:11.37 >> Somebody asked me the other day what my favorite part of my job was, and I said that I get to use all these nerdy interest databases and Excel spreadsheets and data, and I get to use those to help people. So similar to what Elisa said, I love providing support to districts, and I love the data, as well. So it's kind of a nice mix. And the fact that one day never looks like the next ...

00:17:43.71 >> Mm-hmm.

00:17:44.45 >> ... where it's constantly changing, even ... It might just be the data set that I'm working on. It might be that one day, I'm doing a training. The next day, I'm collecting data. The next day, I'm working on public reporting. But it never gets boring.

00:18:00.40 >> Mm-hmm. Yeah.

00:18:01.72 >> Can I pop in another thing?

00:18:04.34 >> Yeah. Yes.

00:18:05.40 >> I really enjoy that I'm always working to make it better, make the ...

00:18:10.75 >> Mm-hmm.

00:18:11.17 >> ... data better, the process better. And so like Amy said, it's never exactly the same because you and others that you're working with are always trying to advance it forward for your state.

00:18:28.39 >> Right. It seems ... Yeah.it would be impossible to have everything be 100-percent perfect. There's always room for improvement and especially as there's new requirements and changes that come through. It's always going to be evolving. And, Amy, what would you say that you're grappling with right now in terms of data quality challenges, and how are you working to address them?

00:18:55.54 >> I think with the data system that we have, they just made updates to the data system. And any time there's an update, there is a change that causes the directors of special education in the state to sort of panic.

00:19:15.09 >> Mm-hmm.

00:19:15.96 >> Because they're not data people. They're special ed people. But they're having to review data and report data, and we have such a turnover ...

00:19:27.28 >> Mm-hmm.

00:19:28.19 >> ... in directors of special education every year, even in the middle of the year, that it's hard to keep the training up. We record the trainings, but sometimes people don't even know that the trainings are out there because they're that new.

00:19:44.48 >> Mm-hmm.

00:19:45.35 >> So I think it's keeping up with the updates in the system and communicating to everyone, especially the new people that you don't ... It's not as hard as it seems.

00:19:58.97 >> Right.

00:19:59.11 >> It's not as scary as it seems. Please don't panic, because I'm here to help.

00:20:03.37 >> Mm-hmm. Alisa, what about you?

00:20:06.12 >> I think also the turnover really is the greatest challenge right now because we've got turnover not only in special ed directors every year. We've got turnover in data enterers.

00:20:20.92 >> Mm-hmm.

00:20:21.22 >> We've got turnover in those registrar folks. And they all contribute to either the high quality or not-so-high quality of data. And they're not just learning how to submit information to the state. They're also learning how to enter that information into their student information systems because LEAs in Idaho have access to use whatever student information system they want ...

00:21:00.96 >> Mm-hmm.

00:21:01.18 >> ... use whatever platform or IAP software they want. And while we have now a lot of folks on the optional IAP software, it all interacts with those student information systems differently. So there is a lot of learning going on, and at the state level, we don't necessarily have the capacity or ability to address all of those learning situations with that student information system. So we have lots of calls come in of, "Could you tell me how I enter this piece of information into my student information system?" It's like, "Well, I can tell you how to submit it to the state and what that's going to look like. But I can't tell you these other pieces because I don't know your software system."

00:21:59.22 >> Mm-hmm.

00:21:59.51 >> I don't know what version you have, and so there is just a lot learning that's going on in each district and that the level of turnover plays a huge, huge amount into that piece. And while you're going through and doing all this training annually on data quality, you have to make sure that you're meeting all of those folks at the level they're at so you're addressing those new to the data recording, and then those who have been in for 15-plus years. So ...

00:22:38.54 >> Yeah.

00:22:38.68 >> That's probably highest level.

00:22:41.80 >> Hmm. So it's similar, I think, in both states in terms of the turnover, your data systems' complexities, training new folks, all of that. So finally, I just wanted to know if there's something you want to talk about you're working on now or something that you're planning for in the future related to improving the quality of your state's IDEA data that you'd like to share with us. Alisa, do you want to kick us off?

00:23:09.15 >> Yeah. I'll jump in. So one of the things that we're trying to figure out, get better at, is all of that public reporting and the ... how that relates back to the data quality is really, if people understand what it's about, they're more likely to put a higher level of importance onto recording good-quality data.

00:23:37.76 >> Mm-hmm.

00:23:38.40 >> So we're trying to make sure that not only are we addressing those 508 accessibility requirements and those basic federal reporting requirements that we're also making that information more understandable and usable for our public. And negotiating those criteria can be pretty tricky because you try to provide it out in more infographic layouts. Then, you're potentially running into those 508 accessibility issues.

00:24:15.08 >> Mm-hmm.

00:24:15.44 >> And so we're just trying to get better, trying to make it more, like I said, understandable, usable and valuable ...

00:24:24.87 >> Yeah.

00:24:24.97 >> ... to not just our teams at the state but at the local level and to families, as well.

00:24:32.40 >> Mm-hmm. Amy, what about in Kentucky?

00:24:34.99 >> So we're working on updating how we collect data and data reports from districts, just trying to make a more ... make it more efficient and ensure that we're getting accurate data like indicators 11, 12 and 13, for example. We collect those directly from the LEAs, and training around that has been difficult, again, because of turnover. And it's ... It can be confusing to them. So we're working on that. And something else I would like to do is, for districts who don't make determinations, do a data retreat.

00:25:09.97 >> Mmm.

00:25:10.47 >> It's something I've been thinking about for a while, and I feel like it would help people understand their data and be able to make changes accordingly instead of just trying ... looking at the overall data or even not even looking at data at all ...

00:25:26.42 >> Mm-hmm.

00:25:26.66 >> ... to make changes. I feel like if we could do sort of a data dive with them and let them disaggregate it and see what's going on within the numbers, I feel like that would help districts a lot.

00:25:39.20 >> Yeah, definitely. That's sounds like a great idea, and I would love to hear more about it and hope that happens.

00:25:45.43 >> Me, too.

00:25:47.18 >> Well, thank you both so much. Really appreciate you being on and sharing your experience and knowledge and expertise in this just ever-changing world and role. And it sounds like you have a lot of great stuff coming up.

00:26:04.37 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at IDEAdata.org.

View Details

Resources

State Performance Plan and Annual Performance Report

MDE: Introduction to Data Use & Action Process

00:00:07.34 >> Hey. It's Amy, and I am so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.76 >> Hello. Welcome to a new episode of "A Date With Data." I am Amy Bitterman, and today I am joined by Jessica Brady, Julie Trevino and Aaron Darling, who are from the Michigan Department of Education. They are going to be describing a data use and action plan process that they have been implementing with their districts. So let's start off by having each of you say a little bit about yourselves and your role. Jessica, would you like to go first?

00:00:52.60 >> Thank you. My name is Jessica Brady. I work in the Michigan Department of Education Office of Special Education. I work in the performance reporting unit, which is responsible for all things special education data, so the SPP/APR and 618 data, also responsible for all of the monitoring of the Individuals with Disabilities Education Act and the Michigan Administrative Rules for Special Education and then corresponding technical assistance.

00:01:22.72 >> Okay. Thanks, Jessica. That's a lot that you're handling. Julie, what about you?

00:01:29.73 >> Hi, Amy. My name is Julie Trevino. I am the SPP/APR coordinator. I've worked for the Office for 15 years now, kind of worked my way through different positions, but I worked with as a data analyst initially, and so ... but pretty much the entire 15 years all related to anything SPP/APR and significant disproportionality, working in the performance reporting unit under Jessica Brady.

00:01:59.77 >> Great. Aaron?

00:02:01.97 >> Yeah. Thanks. I'm Aaron Darling. I also work with Jessica in the performance reporting unit, and Julie. And I started off about 7 years ago supporting the Special Education Advisory Committee, so SEAC.

00:02:16.07 >> Mm-hmm.

00:02:17.03 >> And then I also assist with [Indistinct] Pro and coordinated early intervening services. And now I'm also taking on the data use and action process, so I coordinate some of that work, as well.

00:02:31.46 >> Great, and that is a great segue into the topic of our conversation. I heard through another TA provider about this really cool process that Michigan was implementing, and I spoke with Jessica, and it just ... It's so fascinating and such a wonderful process that I'm so excited to hear more about it. So if you could just start off by just describing to us what this process is, what it looks like, how it works.

00:03:01.71 >> Yeah. I'm Aaron. I can take that one. So I'm the long-winded one today, but ... so bear with me. But this is a four-phase, eight-step process, and the visual logo is in the shape of a figure eight. And so our prepare phase is the first phase, and it has our first two steps. The first step is to organize our data, so we have folks inventory all the data sources that are at their disposal. We take time to orient to the file structure, which is mostly Excel for us. We walk them through the different data definitions, the purpose of each data source, when it's collected, how timely it is. Then we spend a little time on data visualization with charts and graphs, as well. Then step two is, assess data quality. And so we know that not all data are created equal, so we all subconsciously judge data quality, but this in our process is an intentional step intended to identify quality issues so that we make sure we use quality data as a foundation to drive our decision making moving forward in the process. Then we get to move on to the inquire phase, so step three is conduct data analysis. And we teach five really simple but powerful methods of data analysis that should be conducive to almost any learning level. So we have trends to segregation, gap analysis, comparison and outlier identification. And those five typically get us everything we need in order to move through the process. Then we move on to step four, which is determine actionable causes. And so in this step, we focus on finding our root causes, and we want to be sure that schools are able to influence and control those. In education, there sometimes lies things outside of our control like zip code or socioeconomic status, but we try to direct folks to focus on identifying challenges that are rooted in the data and are truly actionable, meaning that if we can address them properly, student results should get better. Then we finally move to our planning phase, and step five is develop people smart results. So many folks in our field know what it means to write a smart goal, but not everyone takes time to focus on being people smart. And we know as humans we're all kind of resistant to change. Plus, most of us have seen many initiatives come and go over time, and folks often say, "This, too, shall pass." But we do not want data use to be one of those things. So we devote time to engaging with and bringing people along who have boots on the ground that will make a change to improve student results. Then, after we consider our people, we move to step six, so we identify strategic activities and plan. In this step, we take time to consider the context and culture of each school, what it means to be truly evidence based and what interventions are available that fit the specific need identified by the data and then finally how that intervention will be implemented with integrity and fidelity over time. So we intentionally plan action steps for 30, 60 and 90 days including who will do what by when and what the expected outcome will be. Then we move into the act phase, and that starts with step seven, to implement your plan with integrity. We know that context differs and that people often customize the intervention or practice to fit their unique needs. That's why we use the word integrity here, so as long as the core of the intervention, including things like frequency and duration, are intact, we can progress monitor the integrity of the intervention and the student results over time. Then we hit step eight to evaluate progress. So finally, as we collect new progress data, we take time to go back through the entire process as we organize our new data, assess this quality, conduct analysis and so on in order to make needed adjustments over time to ensure student success. So in a nutshell, that is the eight-step figure eight data use in action process.

00:07:18.10 >> Thank you so much. Such a robust, well thought out kind of plan, do, study act process, really interesting and need to hear about. Can you say a little bit more about the reason maybe why districts will be going through this process? Is it for significant disproportionality? Is that one of the reasons? Could it be other reasons, too? Just a little bit about how you're determining which districts are doing this.

00:07:45.89 >> Here, Amy. I'm going to take that question. So this process is intended really to build capacity to improve outcomes for students with IEPs. It can be really utilized, excuse me, at the state ISD or what we call in Michigan the member district level. It's evidence based and is intended to be sustained in each level within our system. And so we're really looking at to improve student outcomes, and it's compatible with other initiatives in Michigan, so our top-10 strategic plan and also our MiCAP. And so really what we're trying to do is look at all of the data because we know that at the state level, we have really broad data, so state performance plan data and other data that is really at that aggregate level ...

00:08:34.97 >> Mm-hmm.

00:08:35.31 >> We're trying to get districts and ISDs to look within their system at the data that they have and really make informed decisions on where root causes are ...

00:08:46.30 >> Mm-hmm.

00:08:46.90 >> ... to inform their systems in order to make actions and sustainable actions to increase student outcomes.

00:08:54.29 >> Great, and why did you decide to kind of develop and implement this process?

00:08:59.64 >> So we decided to develop and implement these processes because as we were looking at the state-level data, we saw that Michigan was doing really well in those areas of compliance.

00:09:11.84 >> Mm-hmm.

00:09:12.67 >> And we wanted to move forward and improve those results areas, so graduation, drop out, assessment, et cetera. We knew that that is more of a problem that we have with not just in special education but also looking at general education, and that's where most of our students are in with educational placement. And so working together, regular education, general education and special education, how can we work together to improve graduation, drop out ...

00:09:48.78 >> Mm-hmm.

00:09:49.35 >> ... assessment, those results indicators that are really hard to improve on a daily basis and working with our gen ed counterparts to improve that in an evidence-based way? We know that when typically we do professional development where people are setting and getting, those activities are not sustainable. This really is a process that the Office of Special Education, it takes a year ...

00:10:16.84 >> Mm-hmm.

00:10:17.20 >> ... four workshops over 2 days, and in between those workshops, the participants have homework to do where the Office of Special Education provides support during those homework sessions to gather data to make sure that these individuals have the support that they need in order to accomplish the goals and come back so that it is sustainable and that the next year, when we are not in those four workshops, the ISD member district can go and do this process again. And so it's sustainable in that we're working on improving those student-level outcomes and reaching the students within our districts.

00:11:01.74 >> Mmm. Well, it's great to hear about that general education component and how they're partners in this work, because we know that that's sometimes often a key to success, when you're kind of trying to make change in special education. It has to be a partnership across education to make that happen, so that's great to hear. I would imagine that this is a pretty big undertaking. Are there challenges that you've encountered along he way, and if so, would have been some ways that you've tackled them?

00:11:33.93 >> Well, I'll jump in on that question, Amy. Yes. [Indistinct] some challenges, definitely. I guess I'll start by saying that scheduling and setting aside ... as Jessica mentioned, setting aside four 2-day work sessions throughout the school year, that's a beast in itself, as you're probably well aware that educators rarely have dedicated time to problem solve and often get swept away by just the whirlwind of the day-to-day challenges.

00:12:05.42 >> Mm-hmm.

00:12:06.19 >> In fact, we actually use a river rafting analogy in ... when we're presenting and through the process to kind of describe how we're kind of feeling because we kind of feel like we're getting in the raft with our intermediate school district and our district partners to navigate our way through this process together. And as you can expect, sometimes the waters get rough and turbulent because we really run into various levels of understanding. As an example, some folks we work with can easily sort and filter through that ... those Excel files like in the early stages that Aaron described about analyzing the data ...

00:12:44.54 >> Mm-hmm.

00:12:44.77 >> ... and sifting through it. And others have no idea what even filtering is.

00:12:50.03 >> Mm-hmm.

00:12:50.57 >> So you have those different, varied levels of skills, and we help them with that because that's okay. We kind of understand that everybody is coming to the table with different skill sets. And then the other thing is that other times, the water really seems to get rough as districts question and resist state-level data.

00:13:09.07 >> Mm-hmm.

00:13:09.78 >> Right?

00:13:10.22 >> Mm-hmm.

00:13:10.54 >> So those are tough conversations to navigate through. But we really get through them by leaning into our common purpose of improving results for students.

00:13:21.22 >> Great, so through all of that, I'm really excited to hear, now that I think that it's been a few years that you've been working with districts and they've been implementing this, what are some of the changes that you have seen that possibly are in part due to going through this process?

00:13:38.34 >> So it's been exciting, so at the end of the year, each in-service distribution data action team that's participated gets the opportunity to report back on their progress monitoring data, and more often than not, we're actually seeing improvement in student outcomes.

00:13:54.87 >> Wow.

00:13:55.04 >> That's like a big win, right?

00:13:56.53 >> Yeah. That's the goal, right ...

00:13:58.30 >> Right.

00:13:58.40 >> ... ultimate goal.

00:13:58.97 >> And it is, and it's kind of rejuvenating. It gets us excited. We work hard all year long, and that's ... At the end of the year when we hear back, that's the most exciting time for us. And we actually had even an in-service distribution who requested that we teach them and support them in the process so that they could work with each of their districts within their ISD.

00:14:22.18 >> Mm-hmm.

00:14:22.50 >> And really, that's the kind of success we want, right?

00:14:25.01 >> Mm-hmm.

00:14:25.30 >> We wanted the capacity to ultimately improve outcomes for students. And I think much of the success has come from really setting aside dedicated time to understanding the data. We collect so much information, so much data, and sometimes we just end up admiring the data, right? I think we've all been there, right?

00:14:44.71 >> Uh-huh.

00:14:46.50 >> But we rarely take the time to really understand the complex root causes and the actions we can take to address them. And this process has built-in time that gives the focus time to look into those root causes and planning to take actions. The process, I think, also helps to build trust between the state, which is us, and ISDs and member districts that we're partnering with, and that's been really super beneficial. Approaching them, I think, from a different angle, not just monitoring and coming down on them ...

00:15:19.22 >> Mm-hmm.

00:15:19.45 >> ... it kind of feels like, probably, to them, but actually working together to help them understand and act on their data. And it's really building that psychological safety that we talk about, that partnership that we're in this together.

00:15:32.85 >> Mm-hmm.

00:15:33.06 >> We're on that rack together. But I guess another two things, too, also to mention is specifically to speak on the strategies we feel have contributed to the success of this whole process ...

00:15:44.73 >> Mm-hmm.

00:15:45.29 >> ... are the two parts that Aaron did a really nice job of describing in the process, which are not typically included in other improvement efforts ...

00:15:53.26 >> Mm-hmm.

00:15:53.51 >> ... and that's ... that, too, which is assessing the data quality ...

00:15:56.62 >> Mm-hmm.

00:15:56.88 >> ... right? Because that's the foundation for all the decisions that are going to be made. And then also being people smart in the developing those people smart results. So I think those are vital, vital in our process in the decision making and sustaining change over time. Those are the things that have been big to contributing to the success of this process.

00:16:19.59 >> Great. So what do you have coming up for this work moving forward? What are the next steps?

00:16:27.11 >> So we are planning on ... We're going to continue building capacity with our ISDs. We see that through this process, we have seen improved student outcomes.

00:16:37.81 >> Mm-hmm.

00:16:38.21 >> So it's worth the effort that the Office of Special Education has put on for this data use and action process. Our ISDs have seen the effort that they put in as worthwhile. We ... At all of our meetings with our ISDs, we have ISDs speak about this, and they see the improved outcomes. They enjoy doing this work. They know that it's hard work. But they do see the benefit. And so the Michigan Department of Education Office of Special Education will continue to build our capacity. So every year, we add more staff to our team in order to build out and add more in-service distributions to the process.

00:17:26.48 >> Mm-hmm.

00:17:26.77 >> So as we're adding more ISDs, we're also supporting the ISDs that we had the previous year.

00:17:32.00 >> Mm-hmm.

00:17:32.57 >> We also now have ISDs that they are in their third year of needs intervention for determinations.

00:17:40.70 >> Mm-hmm.

00:17:41.13 >> This is a process that we tell them that they are going to be participating in because we've seen such growth in ISDs and member districts ...

00:17:51.02 >> Mm-hmm.

00:17:51.27 >> ... and improving student outcomes, so this is an activity that we are having them participate in for the year as a result of being in needs intervention for the third year. We strongly believe in this process, and so with a dedication of staff, we and ... putting in ISDs who are volunteering on a basis that we cannot sustain at this point.

00:18:15.93 >> Mm-hmm.

00:18:16.14 >> And we're saying, "We'll have to get you on a following year" ...

00:18:18.88 >> Mm-hmm.

00:18:19.71 >> ... because how many ISDs really want to join us in this process because of all of the data that we've shown them throughout the 3 years that we have done this? Now, I will say that this has to be done in person.

00:18:34.20 >> Mm-hmm.

00:18:35.23 >> We have done this, and we have ... to do this virtually. It did not work because the collaboration that needs to take place in person, we have also done it just at the ISD level where ISDs have not brought in member districts. It did not work in that instance, as well, so it really is a process that we've had some trials ...

00:18:58.42 >> Mm-hmm.

00:18:58.66 >> ... and then failures because during COVID, we tried that, like I said, virtually. You cannot do this process virtually. You must do it in person.

00:19:07.32 >> Mm-hmm.

00:19:08.02 >> It really is a process where you have to be fully engaged.

00:19:12.27 >> Yeah.

00:19:12.41 >> And we know sometimes when you're virtual, you're not fully engaged. You need to be with the people, hands-on, with the data, wrestling with it and getting your hands dirty. when we did it with just the intermediate school districts, it didn't work because they didn't have enough data to go down deep enough into that student level. So you need where your students really are. And so those are some heads up to, if people are ... want to do this process, you need to have deep enough data in order to make the change because you are picking evidence-based practices to implement with students.

00:19:52.43 >> That's so exciting, that you have more districts that want to participate than you can even really handle right now. So that really shows the level of interest and how it's been demonstrated to really be a successful process. And I'm sure others that might be listening to this episode would love to kind of get more information for ... dig in a little more to see how all this works. Do you have on your website more about the process?

00:20:19.29 >> So, yes, we have information on Michigan Virtual University. We have a little course.

00:20:25.08 >> Mmm.

00:20:25.76 >> And then we do have information on catamaran about a little information about the data use and action process.

00:20:34.06 >> Okay, great. We can put links to those in the description of the episode so folks can check that out. Well, thank you all so much for sharing about this wonderful process. It's so thrilling to hear about something that seems to be working and that districts are interested in doing. So many times, you hear about things that, especially in situations where the state is kind of requiring them to do it, it's their resistance. But the fact that they really want this and it seems to be working is really cool. So thank you all for being on the podcast.

00:21:11.16 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at IDEAdata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.64 >> Hello! Welcome to "A Date with Data." I am Amy Bitterman, and this is a very special episode because I'm so excited to be joined by two of my wonderful IDC colleagues, Terry Long and Chris Thacker. And they're both IDEA TA providers among wearing a number of other hats, and they're going to be talking about IDC's Edit Check Tools with us. So welcome, Chris and Terry. Thank you for being here.

00:00:51.12 >> Thank you for having us.

00:00:52.56 >> Hey, good to be here, Amy.

00:00:54.50 >> So the Edit Check Tools, I actually back in the day worked with Terry on them many years ago. So we've had them for quite a while now but definitely have been making improvements and changes, but for those maybe who aren't familiar or haven't used them before, can you just give us a brief overview of what the Edit Check Tools are and how states can use them?

00:01:15.60 >> Sure. Chris, do you want me to jump in?

00:01:17.73 >> Yeah, I'm going to just defer to you in most of them, Terry, and I'll just follow up if I feel the need to add anything.

00:01:23.55 >> Sure, thank you. Okay, so IDC does have a suite of Edit Check Tools for SEA EDFacts files and LEA EDFacts files. We also have a couple of tools that we aren't going to focus on a whole lot today, but I do want to mention that we have Edit Check Tools also for MOE CEIS data and dispute resolution data that are submitted via EMAPS.

00:01:49.89 >> Mm-hmm.

00:01:50.31 >> So the suite of tools that we're focusing on today will be the ones that are the tools that help us visualize and review the EDFacts files that are submitted for childcare and education environment, exiting, discipline and personnel data, so all of those 618 data files.

00:02:20.70 >> Okay.

00:02:22.40 >> And I would just add that both Terry and I are former state data managers, so this is probably a project that's been near and dear to our hearts because it's something that we saw as a need in the field before ever coming to IDC. And I think Terry has led the way on developing these tools and done an outstanding job in making it as easy as possible to get the data into the system and to be checked and to make sure everything is accurate.

00:02:49.97 >> Yeah, these were definitely tools that when we were Part B data managers together, Chris, that we saw a need for way back then, and we always wanted to be able to make them but it seemed like we never had time to do that. So I was so grateful when we started working as TA providers with IDC, and they gave us the opportunity to make these tools a reality.

00:03:15.82 >> Yeah, these are really wonderful tools and are so helpful to states so they can run their files through and see if there's any issues or anything that might be flagged before they have to go into EdPass and actually submit, so they can get those addressed quickly. And then I think one of my favorite parts, like you mentioned, is that data visualization piece where those files are not easy to read for anyone in and of themselves. So these tools have the capability and are able to aggregate and display all of that data in really user friendly ways.

00:03:58.34 >> Yes, so some ...

00:03:59.48 >> And I like, too, Amy, the way that the tools will present those data displays in alignment with how the EDFacts file specifications will group those different items together. I think that's very helpful, and as you noted, the raw EDFacts file isn't a reader friendly document at all. And by having those displays built into the system, it makes it real easy to share that information internally with your state-level staff, and then you could even use it for external purposes as well once you get it vetted.

00:04:35.08 >> But it is an easy way that, adding your EDFacts file data in, you literally ... To put that data in, you literally copy and paste a copy of your EDFacts file, whether it be child count, the FS002 or 089.

00:04:52.14 >> Mm-hmm.

00:04:52.76 >> You actually copy and paste those into the tool, and there are a lot of handy formulas and some macros in the tool that then allow the data to be displayed into some real user friendly tables as well as some edit check messages display that are easy to read and include hyperlinks to take you right back to the original file to find the location, the row where an error may be occurring. So it ... The tools really are a great way for EDFacts coordinators to communicate with their Part B data managers as far as reviewing the data in the files. It just makes it so much easier than just taking that raw file and then having to put headers on it and do a lot of other more complex manipulation in Excel to review the file.

00:05:54.81 >> Right, and that's, I think, something you mentioned among many things that really a lot of states like is, when they are getting those error messages, it can be hard to track down exactly what data the error message refers to, what the problem is, and this tool, I think, really helps to better pinpoint where the problem is in the data.

00:06:17.76 >> It does. It does. It actually has hyperlinks that will take you back to that version of the file that you had pasted into the tool so you can identify exactly which row or rows may be involved potentially in an error or something that just needs to be reviewed to make sure it's accurate.

00:06:38.15 >> And like you said, too, it's not ... There's really no work involved in using this tool. It's just a matter, like you said, of copying and pasting your EDFacts file right in, and then the rest happens magically due to Terry and Chris' amazing programming. So it is a very, very cool suite of tools. So talk a little bit about, now that we are moving to using EdPass, are there any changes that you have needed to make to the tools, and if so, why and what have those been?

00:07:11.84 >> Well, we definitely ... When we learned about EdPass, we had to stop and ask ourselves, "Well, these tools are doing the same sort of work that is now completed in EdPass as far as doing edit checks to the data prior to submission." So we spent some time really thinking and talking as a group at IDC about whether ... Are these tools now obsolete? Do we not need them anymore, or do we need to go ahead and keep them up to date, keep them available for state use? And after a lot of discussion, we decided that, yes, indeed, we would want to keep them in place because we know that for EdPass, it's only available about 6 weeks prior to the submission deadline for the files. And so having these tools out there and available at any time, a SME can pick up, if they have their exiting data, for instance, available early ...

00:08:20.27 >> Mm-hmm.

00:08:20.74 >> ... and EdPass is not yet open for that submission, they can pick up that tool, the SEA tool, paste in the SEA data, or the LEA tool and paste in the LEA data to review what's happening, run those same sorts of edit checks that are used in EdPass but be able to do it prior to the opening of that EdPass window. So we decided that having that, allowing for that flexibility for states was important. So IDC decided to move forward with continuing to update and maintain these tools and improve the tools.

00:08:57.00 >> Mm-hmm.

00:08:57.24 >> So one of the improvements that we've done is to add ... very recently add our year-to-year ...

00:09:03.64 >> Mm-hmm.

00:09:04.65 >> ... check capability in the tools.

00:09:07.00 >> I might add, too, that Terry mentioned the fact that the EdPass window is only open for 6 weeks or so, but it's also been pushed back much later than what it was previously under the old system so that it will be much later in the year that you would, as the data manager, have the opportunity to go in and check the data the way the EdPass is checking it were it not for these tools. Right now the tools are available for download. They can ... As soon as you get the data together there at the state level, you can start creating those EDFacts files, putting them into the system and doing some checking long before the EdPass system is ever up and running. So it gives you that ability to know sooner rather than later how well your data are coming together.

00:09:56.89 >> Yeah, and that year-to-year comparison, too, is really important, especially now because it's not ... It was not part, at least for the child count net environment submission in EdPass. Those checks were not automatically run in EdPass, so having the Edit Check Tools to do that is incredible.

00:10:16.32 >> Yeah, I think the EdPass right now doesn't have the historical data in it to do ...

00:10:20.20 >> Yeah.

00:10:20.30 >> ... those year-to-year comparisons. I believe it will at some point ...

00:10:24.09 >> Mm-hmm.

00:10:24.30 >> ... down the road. This tool where Terry updated it for the child count, and it will be updated for the other tools as well, gives that ability to check your current data to a prior year data. Usually it would be the most recent year, but you would have the flexibility, if ...

00:10:40.37 >> Mm-hmm.

00:10:40.62 >> ... you chose to add in a different prior year to look at it more historically if you were inclined.

00:10:46.37 >> Mm-hmm.

00:10:46.62 >> Yeah, if you really wanted to take a look at, say, what your data looked like prior to COVID ...

00:10:52.18 >> Mm-hmm.

00:10:52.48 >> ... during COVID, after COVID, that kind of thing, you can ... It's up to you which years you compare in the tool. So you can use it for that straight year-to-year comparison that OSEP uses for looking at data quality, but you can also use it for other analysis purposes as well.

00:11:13.11 >> Yeah, think outside the box. There's a lot of uses, I think, for this tool other than the standard way that folks might think about it. I know IDC has provided technical assistance to states in using these tools. We've also done other events. Can you talk about how IDC supports states with these tools?

00:11:37.25 >> Sure. We can definitely, at any time, you can contact your IDC state liaison and set up a time to walk through these tools on an individual basis, if you'd like, anytime that service is available to you. Also, I think, Amy, you're referring to getting some groups of states together to spend some extended time using and reviewing data in these tools.

00:12:08.72 >> Mm-hmm.

00:12:08.95 >> We've been able to do that through IDC's Hands-On Learning Academies. We've had two of those now. We had one about a year and a half ago and then one just this past August, and we were able to bring a few states together into a room for a couple of days to spend time. We asked them to bring their EDFacts file data with them, and we had copies of each of the IDC edit check and data disability tools available to them, and we just literally spent time plugging in their state and LEA-level data into the tools and just really spending time examining the data in those tools, looking for any errors that may have existed.

00:12:59.27 >> Hmm.

00:12:59.36 >> Most of these files that they were bringing to us, of course, were files that had already been submitted ...

00:13:04.33 >> Mm-hmm.

00:13:04.58 >> ... in EDFacts, so they were ... had already been vetted that way. So most of them were error-free, but it did give them a chance to see these data displayed in the data tables, and like Chris was saying, we have all of the subtotals that are represented in the EDFacts files as well as all of the category sets. So each of the subtotals and category sets that are required in the EDFacts files are displayed as data tables in the tools, and it gave these states time to really spend a good amount of time that they don't ... may not typically have when they're sitting in their offices, moving through their crazy daily schedules to actually stop and pay attention to the data that they're seeing in this table format.

00:13:58.26 >> That time component that Terry just mentioned there, I think, is very important. Something that I was thinking about as she was going through this, at these Hands-On Learning Academies, it does give the data manager or the EDFacts coordinator or perhaps an SPP/APR lead, ever ... who comes from your team an opportunity to work together and look at that away from the office where they're not as likely to be interrupted on things, and it also ... There's a real advantage at these groups of having other states there in the same room with you that might present ideas or situations or circumstances that might not have come up in your state, but you get to hear it from a different perspective or look at it from a different angle, and that turns out to be very, very helpful. So the two that we've had so far, we've been well pleased with the participation of the staff from the various states who have attended and the feedback that we've gotten from them, and we hope to be able to do some more in the future. In addition to those events, I know that at some of the various conferences that we've put on, like our Interactive Institutes where we will have tools that we share ...

00:15:04.46 >> Mm-hmm.

00:15:04.82 >> ... We shared this at a couple of institutes in the past and gotten a lot of participation from states. I know I work with a number of states as a state liaison, and many of those states have asked for explain on how to use these tools, and I've sat with them, as Terry described a while ago, and gone through the process of loading the information in there and to look for errors and to diagnose problems and issues, and it's a real great, useful tool, I think.

00:15:33.18 >> Yeah, so if anyone out there listening needs any support, wants a tutorial, has questions, feel free to reach out to your state liaison, and we will connect you most likely with either Terry or Chris or someone else that knows these tools inside and out.

00:15:49.68 >> And we do try to jump in when you make one of those requests. We know that you're working on that data right at that time and that you typically need an immediate response. So we try to be super responsive when you ask for assistance with the tools. We try to jump right in there and schedule something within the next few hours or the next day or two to talk with you about these tools and especially when it's nearing data submission due dates.

00:16:21.08 >> Mm-hmm. Yeah, when it's crunch time.

00:16:23.53 >> Yeah.

00:16:25.33 >> So what's going on now related to these tools? Is there anything new that you're working on, something coming out in the near future?

00:16:34.85 >> Well, like Chris said, we recently, with the child count that was due back in August, we had updated the SEA tool to include that year-to-year check ...

00:16:49.34 >> Mm-hmm.

00:16:49.55 >> ... Since OSEP was asking states to go move forward with doing a year-to-year check even though one is not available in EdPass at this time because there is no historical data there. There's only the 1 year that's just ... was recently submitted, 1 year of child count and education environment data. So that ... We made that change to the child count tool, and now we're moving forward with adding the same year-to-year check also for the exiting tool that ...

00:17:21.99 >> Mm-hmm.

00:17:22.67 >> ... reviews FS009 ...

00:17:25.60 >> Mm-hmm.

00:17:26.14 >> ... EDFacts files, also the discipline tool, adding year-to-year check to that, and that handles all six of the discipline EDFacts files, and the year-to-year check will also be added to the personnel edit check and data display tool ...

00:17:42.65 >> Mm-hmm.

00:17:43.44 >> ... that ... And that handles your FS70, 99 and 112 files.

00:17:51.52 >> I would also add, though, this is new, as it the comparison, but the fact that we have the LEA version ...

00:17:58.30 >> Mm-hmm.

00:17:58.62 >> ... of the tool is very, very helpful and fairly recent in comparison to the state tool itself, and the LEA tool, a really good function or feature that it has, it displays or does those data displays just like the state tool does, but it will do it for individual LEAs.

00:18:15.18 >> Mmm.

00:18:15.39 >> And that's something that you can share with your monitoring team or others that ... in the office that might have specific interests about a specific LEA that there's concern about or just part of your routine risk assessment that you do. You can look at individual LEAs and look at it in those data display options to get a better feel for the data. So I hate to overlook the significance and importance of that addition of the LEA tool ...

00:18:42.69 >> Mmm.

00:18:42.97 >> ... not too terribly long ago itself.

00:18:44.81 >> Yes, those are much newer tools, too, Chris. Thank you so much for reminding us about those.

00:18:51.11 >> Mm-hmm.

00:18:51.70 >> The LEA tools do give states a way to dig down into their LEA-level data and can be used by monitors as they prepare for monitoring visits and for the other ... all kinds of things that ... reasons why you need to pay attention to your LEA-level data. This gives you an easy way to see those data that have been submitted in EDFacts files at the LEA level. I was just wanting to also invite those out there who are using these tools to send any feedback they have ...

00:19:30.52 >> Mmm.

00:19:30.70 >> ... about them, anything that they would like to see, any additions, any functionality additions they would like to have added to these tools. We would love to hear from our state folks out there using the tools and have them give us that kind of feedback, tell us how ... Are there additional features that they want to see in the tools? We would love to hear that kind of feedback from states, and we would definitely take it into account and see if we can make it happen.

00:20:02.97 >> And also share the good news that you might have and how well the tools work for you. The positive comments are always great to hear, too, and it might be something that if states themselves share some value that this has provided them, that other states might see some value in the tool as well.

00:20:21.29 >> Sure, and we do know that some states actually use the tables that are provided, the data display tables in the tools as a part of their requirement to publicly report their 618 data.

00:20:36.43 >> Mm-hmm.

00:20:36.86 >> So we know that that's one of those additional uses of the tools that's out there. So like Chris said, we would love to hear from others about how they're using the tools and spread the news out there. So to their colleagues, give them ideas about ways to utilize these tools in creative ways to ... that help them do their work at the state on a daily basis and also help their LEAs and schools better understand their data.

00:21:11.46 >> Yeah, absolutely. All right. Well, exciting things happening in the Edit Check Tool world, and hopefully those of you who haven't used these yet might give them a try. If, again, any questions or support that's needed, we have Terry and Chris waiting and willing to jump in and provide that support, and I'm sure we'll probably be doing more of those Hands-On Learning Academies and other events in the future. So thank you both so much, Chris and Terry, really appreciate you being on.

00:21:47.67 >> Thanks so much for having us. It was great talking with you, Amy.

00:21:50.98 >> Yes, it was, as always. Take care.

00:21:54.13 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data center.

00:00:24.78 >> Welcome to "A Date with Data." Our guests for today's episode are from the California Department of Education. I am joined by Dr. Jack Brimhall, who is the Interim Associate Director of the Special Education Division, and also Kishaun Thorntona, who is the Education Program Consultant who is also with the Special Education Division, and we're going to be discussing California's significant disproportionality data and systems as well as how the state supports districts that have been identified with significant disproportionality. Welcome to both of you.

00:00:57.53 >> Hey, there. Glad to be here.

00:00:59.47 >> Hello, everyone. Thank you for having us.

00:01:01.65 >> Yes, so just to get things going, if you could each just say a little bit about yourselves and your role in the California Department of Education, Jack, do you want to start us off?

00:01:11.99 >> Sure. Yeah, so I'm Jack Brimhall, and right now I'm the Interim Associate Director with the Special Education Division at the California Department of Education. And one of my roles is the head of the monitoring units, and we have a variety. We have actually five different ... six different monitoring units. Two of those units deal with significant disproportionality, and so we have consultants in those units that work really hard with districts that have been identified as significantly disproportionate. We have another two districts that work with districts ... or two units, excuse me, that work with districts that are disproportionate. And I've been at the California Department of Education for the last 10 years. Before that, I was a high school principal, and before that, I was a high school English teacher, so happy to be here.

00:02:03.47 >> Great. Thank you. Happy to have you. Kishaun, what about you?

00:02:07.26 >> I am Kishaun Thorntona, and I am one of the Focused Monitoring and Technical Assistant Unit Consultants at the California Department of Education, so I'm in one of the two units that Jack described as monitoring districts that are significantly disproportionate. I've been with the department for about 19, 20 years, and worked with districts who were both disproportionate and significantly disproportionate, I would say probably for the last six years, and just excited about the work that we're doing with significant disproportionality and how the Department is monitoring the work to ensure better student outcomes.

00:02:54.06 >> Yeah, awesome. So to get things started, if you could maybe just describe for us how California identifies districts for significant disproportionality, maybe provide some context, a little bit about your methodology, how many districts are typically identified in a year, that sort of thing.

00:03:13.45 >> Well, it's very ... I don't ... We don't do the actual ... "We," meaning the two units, don't calculate, but our data unit calculates ...

00:03:23.42 >> Mm-hmm.

00:03:23.63 >> ... does all the calculations, and it's my understanding it's very ... It's a lot of work. And so they do this with all of our districts, and we have, Jack, probably, what, 1,500 districts?

00:03:35.87 >> Yeah.

00:03:36.17 >> Yeah.

00:03:37.08 >> We have over 2,000 districts in the State of California, so it's big.

00:03:41.81 >> Mm-hmm.

00:03:42.02 >> It's a big job, to crunch all that data for everybody not only so that you can see who's significantly disproportionate, but you have to go backwards and see who's disproportionate first and then track it. So it's a big, big job or our data.

00:03:56.64 >> Yeah, so they do all the risk ratios for every district in the areas of disproportionality. Then, they determine ... After determining the disproportionality for every LEA, those that are disproportionate for 3 consecutive years in the same areas are then identified as significantly disproportionate and then are determined or are placed in our monitoring system. And we have approximately ... Since 2020, we've had approximately ... Well, we started off with probably about 138 ...

00:04:34.27 >> Mm-hmm.

00:04:34.65 >> ... districts that are identified, and it's just been over 110 districts identified every year thereafter.

00:04:40.84 >> Wow.

00:04:41.77 >> Yeah, and right now, to be identified for dispro, to being disproportionate, you would need to have a risk ratio of 3.0.

00:04:51.87 >> Mm-hmm.

00:04:52.51 >> And so that essentially means you're three times more likely to either be that ... have that disability or to be suspended, those types of things. So we use 3.0, and Kishaun can talk about it more, but that's one of the areas in which gets you into significant disproportionality, but then we also have reasonable progress that can help you get out, so ...

00:05:19.45 >> Mm-hmm.

00:05:20.43 >> And our our reasonable progress, we started a couple years ago as a result of our stakeholder, then, we called it stakeholder, Sigda stakeholder group that we've had that we've had that conversation about reasonable progress. So our reasonable progress is a district under a risk ratio of 4.0.

00:05:43.06 >> Mm-hmm.

00:05:43.88 >> And so it has to be an "And." There's an annual 0.5 decrease in the risk ratio, so then they'll be considered for reasonable progress.

00:05:54.98 >> Ah. Would they have to be below 4.0 all 3 years or just the most recent year?

00:06:01.71 >> The most recent 4.0 ...

00:06:03.78 >> Got you.

00:06:04.01 >> ... but have an annual decrease of 0.5.

00:06:07.31 >> Mm-hmm.

00:06:07.58 >> And that's what it is now, but as we continue to build our monitoring system, that might change with the consideration of qualitative data. It's really ... It's a lot of work because there's a lot of districts.

00:06:23.68 >> Yeah.

00:06:24.05 >> And so we want to make sure that we're doing what we are ... we should be doing.

00:06:28.53 >> And one of the things that ... There is a lot of districts, and we spend a lot of time working with those districts because we feel it's important.

00:06:37.35 >> Mm-hmm.

00:06:37.99 >> But it's all rooted in data where our monitoring system is rooted in looking at the data, driving deep into the data to come up with some root causes ...

00:06:49.90 >> Mm-hmm.

00:06:50.26 >> ... for the underlying issue so that ultimately these districts hopefully not only just make progress, but they are no longer disproportionate. And so it's all based in data, and it's also based in implementation science. It's based in best practice. And we have folks that are experts in that area that we have worked with those school districts. So it's exciting. It's hard work but exciting work.

00:07:17.18 >> Yeah. I know. I've heard a bit from Kishaun over the last couple of years about all of the wonderful work you all are doing, and it is. It's thrilling, and I think it's just a model that other states and districts would be curious to hear about and learn from. So once you have your group of districts that you know have been identified with significant disproportionality, how do you then notify them? What data or information are you sharing with them to let them know they've been identified and kind of what those next steps are?

00:07:48.54 >> Every year, we give what we call an annual determination letter.

00:07:52.15 >> Mm-hmm.

00:07:53.16 >> And normally that's released in mid-March, and the annual determination letter is going to indicate what monitoring level the local educational agency is in, and in our state, we have three different levels. We have a universal level of monitoring, which essentially means you've met just about all the indicators, all the targets.

00:08:13.75 >> Mm-hmm.

00:08:14.24 >> Targeted monitoring means you've missed some, so, for example, districts that have been identified as disproportionate ...

00:08:20.73 >> Mm-hmm.

00:08:21.04 >> ... would be in that targeted, and then intensive are the school districts that need the most intensive help and that we provide the most intensive help to. But if you're identified as significantly disproportionate, you would be in that intensive care.

00:08:35.68 >> Mm-hmm.

00:08:36.03 >> It's important to note you can be in the intensive care without being significantly disproportionate, but anyway, that letter explains all that to the school district. It tells them what their care is.

00:08:46.56 >> Mm-hmm.

00:08:47.19 >> If they are significantly disproportionate, if they are disproportionate, it tells them all they need to know and then the next steps. And so once that letter goes out, then we start to actively monitor and work with the school district to come up with a plan. And like I said before, all of our monitoring now is rooted in data.

00:09:07.94 >> Mm-hmm.

00:09:08.35 >> So there's four steps to each plan. There's the looking at the data in step one, and not just one piece of data, by the way. We want to triangulate variety ... a variety of data, so local data, indicator data. California has a dashboard, which is really cool, that breaks down chronic absenteeism data, state performance plan indicator data, so data by The Feds, all of these different pieces of data and qualitative and quantitative data. So it might be that you do case studies and empathy interviews.

00:09:43.18 >> Mm-hmm.

00:09:43.44 >> That's that step one where all that data is looked at, and then step two is where you take that data, and now you start to develop some root causes around the data.

00:09:53.15 >> Mm-hmm.

00:09:53.67 >> And once you get those root causes, then you can start developing a plan that's based on that, and with significant disproportionality, it's difficult because once you're significantly disproportionate, now you have to dedicate 15 percent of your IDEA funds ...

00:10:12.99 >> Mm-hmm.

00:10:13.25 >> ... your special education funds to early-intervening services. And so we need to have those funds tied to activities that are going to support what's in that plan.

00:10:25.69 >> Yeah.

00:10:26.12 >> And Kishaun deals with this all the time, but the plan has to be approved before those funds can be spent, and so we want to have a good plan, but it's also a race to get a good plan so that you can start expanding these funds so that then we can reimburse so they don't lose that 15%.

00:10:43.98 >> And if you have well over 100 of those plans that need to be reviewed and approved, that's a big job, too.

00:10:51.52 >> Yeah, and Kishaun can talk to that because she herself has a number of plans, and then they overlap, right, because you might have some new plans that go 27 months. But then if you're significantly disproportionate again, that's a whole separate set, whole separate pot of money, right?

00:11:09.57 >> Mm-hmm. Mm-hmm.

00:11:10.27 >> So you might have 15 percent of your IDEA funds quarterly intervening services in one plan, but then if you're significantly disproportionate again, now you have a different pot of money. So then Kishaun, if she's your monitor, she's got to track and watch two different plans at once, one plan that's already in the implementation stage while trying to approve a new plan and those new activities toward that new money.

00:11:36.68 >> Yeah, a lot of layers of complexity, for sure.

00:11:41.30 >> And in addition to the ADL letter, after that ADL letter goes out, the annual determination letter goes out, the districts receive that. Then, our contractor and partner, State Performance Plan Technical Assistance Project with the Napa County Office of Education, starts contacting the districts to front load some information through introductory videos and then also around the mandatory webinars that we have. We front load these introductory videos. They're about 10 to 30 minutes in length. And it's for LEAs to understand the why, how and what around significant disproportionality, and the process that they'll encounter as a result of their identification, the front loading of information prior to the actual webinars present or offer information on the federal requirements, identification methodology, technical assistance facilitators and a history of the significant disproportionality work as well as the why the work is important and challenging mindsets. And so it's with these ... with this information is like the start of creating that plan that Jack was just talking about ...

00:13:00.93 >> Mm-hmm.

00:13:01.56 >> ... and really helping them create that foundational piece of knowledge so that they could begin to do all of the required work in creating their plan.

00:13:12.46 >> Great, so kind of what comes next, then, in terms of what the districts need to do and the resources and support that the state and your TA provider are providing the districts?

00:13:29.13 >> So the California Department of Education has created the compliance and implementation monitoring process.

00:13:37.83 >> Mm-hmm.

00:13:38.76 >> And Jack talked about the steps that we have in this process. I think one of the things that's really important to point out in this process is having the right people on the bus to begin with. It's so important to have the right leadership team, so that's normally like the superintendent and their cabinet, across both general and special education. A lot of this work is or all of this work is pre-referral work, and having the partnership and understanding throughout the district is really important. So having the right people on the bus is the start, and then it leads to the creating that ... the Educational Partners Team that is required to then review all the data ...

00:14:32.50 >> Mm-hmm.

00:14:33.36 >> ... that Jack referenced and all the different types of data to come to the root causes. That Ed Partners group is so important because generally they will be the ones that are discussing those root causes or coming down and narrowing down to the root causes that the district feels are contributing factors to significant disproportionality. So I think the biggest thing is having the right people on the bus. One of the other things that are ... We call our monitoring process the CIM, Compliance and Improvement Monitoring.

00:15:07.45 >> Okay.

00:15:07.63 >> Did I say that right, Jack? But that process now requires for LEAs in significant disproportionality to do case study, do a case study. So we look at policies, practices and procedures, and some districts do focus or focus groups or empathy interviews to check their practices ...

00:15:35.32 >> Mm-hmm.

00:15:35.81 >> ... as it relates to policies and procedures, but also that case study is really important and looking at the students that are identified. What is their journey in their educational history? What happened? And to really understand the students that they should be addressing. And so that piece is new in our process this year where it's required, and I'm excited for it because we've seen it done in the past, but it was optional work. But now our ... With our new process, we've made it a requirement.

00:16:11.89 >> Yeah, that's really neat because it seems like the practice pieces is what a lot of states and districts struggle with in terms of how to measure that and how to implement that review, and that's a really innovative way you're talking about being able to, it sounds like, do a really thorough job of kind of checking for those practices and sharing that they are appropriate.

00:16:38.85 >> Yeah, California has really been focused on the qualitative data.

00:16:42.38 >> Mm-hmm.

00:16:43.59 >> Getting that qualitative data in all these different ways is really part of the driving force, as well, as the quantitative data that we've always used in the past.

00:16:54.09 >> Well, you all are very ... sounds like has a data-rich process in place.

00:16:59.50 >> It's ... Not only is it data-rich. Kishaun mentioned it. It's that partner team ...

00:17:06.08 >> Mm-hmm.

00:17:06.52 >> ... and getting, in general, ahead as part of it ...

00:17:09.34 >> Yeah.

00:17:09.47 >> ... because it's a general education plan.

00:17:11.49 >> Mm-hmm.

00:17:11.70 >> These funds and these activities are spending funds towards early-intervening education. It's not special education and what special education funds are going toward general education.

00:17:25.12 >> Mm-hmm.

00:17:25.41 >> So essentially it's a general education plan, and special education students are general education students first, so we feel like it's ... That's a huge part of it, is getting everybody together, the gen ed with the special ed and then using quantitative with the qualitative. It's a process. Its not a sit-and-get. It's a process of improvement.

00:17:50.07 >> I think another piece that I left out, too, is how heavily involved parents are in this process.

00:17:54.98 >> Mm-hmm.

00:17:55.78 >> We require parent input, parent partnership, really, especially on the Educational Partners Team in determining those root causes.

00:18:06.80 >> Mm-hmm.

00:18:07.40 >> It's not districts doing it alone. Parents are involved, parents of the identified group is involved. And so we really are looking at the process in which districts took to get their plans, to create their plans. That's really important in this.

00:18:25.46 >> Yeah, that's a really great strategy that you described, especially with those teams and the composition of them, making sure to include families, and general education has to be at the table. So it sounds like you all have really done some great work in making sure that that is a piece of what the districts are doing. Are there ...

00:18:51.12 >> We're trying.

00:18:52.54 >> I know. Easier said than done, but ...

00:18:54.61 >> Yeah.

00:18:55.17 >> That is the goal.

00:18:56.42 >> It's still definitely a work in progress.

00:18:58.46 >> Mm-hmm.

00:18:58.67 >> We're constantly looking at our process and seeing what works best and ...

00:19:06.37 >> Yeah.

00:19:06.69 >> ... how we could support better, and so we're constantly looking at that, I think. But for the foundation of our system that's in place, but how we go about it, we always want to be reflective of our districts and what we're seeing and what we want to see.

00:19:27.83 >> Mm-hmm. Yeah, and to that end, what do you have coming up next for significant disproportionality in California? Are there any new resources or activities, changes you're thinking about making? What does the future look like?

00:19:46.24 >> We're always doing webinars.

00:19:48.47 >> Mm-hmm.

00:19:49.57 >> And our webinars are determined based on what our districts ... we see the need. MTSS is a huge topic.

00:20:01.03 >> Mm-hmm.

00:20:01.39 >> EL is a huge topic, discipline, so those are things that we focus on as we provide webinars throughout the year for our districts to take part in. We always look at the new articles that come out, the academic articles that come out related to student success, how to support students, how to support districts and staff, how to build out systems. A lot of districts are still in that process of creating their foundation and building on it. So we really look at what districts need, and so we constantly try to provide the webinars, provide the additional resources of reading resources. We partner with other providers that are expertise in certain areas, and so if we see a need of our ... the district needing something that we know there is support available, we offer that support. So that's constantly changing all the time.

00:21:08.80 >> Mm-hmm.

00:21:09.92 >> ... depending on what's new that comes out. I think one of the biggest things that we're doing now is really focusing on which ... the implementation piece.

00:21:19.60 >> Mm-hmm.

00:21:20.26 >> That is huge, and we've talked about that in the peer-to-peers for months now ...

00:21:25.29 >> Yeah.

00:21:25.74 >> ... really looking at implementation and focusing on implementation. How can we support that implementation piece? That's huge, I think, moving forward.

00:21:35.10 >> Yeah, that's definitely, I think, what we're hearing from a lot of states, is they sort of are at a place where they've been identifying districts. They feel like the methodology they have down and notifying the districts, but then how do you address the issue? What are some of those practices that really seem to be effective, and are districts really implementing them with fidelity, so that evaluation and monitoring piece, as well?

00:22:02.47 >> Yeah, that part is so hard because every district is so different.

00:22:05.37 >> Mm-hmm.

00:22:06.31 >> The demographics, the size, the ... and every district is so different, and so trying to see what works, what works best is really difficult.

00:22:17.70 >> Right, because what might work in one context may not ...

00:22:21.09 >> Absolutely.

00:22:22.11 >> ... in another. Well, thank you both so much. You shared so much wonderful, interesting information. I learned some things, and hopefully others that are listening have, as well. So really appreciate you coming on and telling us your story from California and how this has all been going.

00:22:42.17 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode context. Or connect with us via the podcast page on the IDC website at IDEAdata.org.

View Details

Resources: Indiana Department of Education, Special Education Results Accountability website - DOE: Special Education (in.gov)

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.61 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.45 >> Hey, it's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education stuff who, just like you, are dealing with IDEA data every year.

00:00:19.60 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:25.01 >> Hello. I am Amy Bitterman, and this is IDC's "A Date with Data." On this episode, I have with me Angela Vaughn, who is the assistant director of Monitoring & Compliance from the Indiana Department of Education's office of special education, and she's going to be talking to us about Indiana's annual LEA determination process. For those who aren't familiar, IDEA requires states to make determinations annually about the performance of each LEA. There is variability in terms of how states use the data elements in the determinations, how they calculate determinations and also how they're applying criteria for the different determination categories. And we're really excited to hear about how you're doing this in Indiana, so welcome to the podcast, Angela.

00:01:12.54 >> Well, it's great to be here. I've heard many wonderful podcasts before, so this is a little intimidating after seeing the great, incredible people and the work that's being done and being part of this, so thank you.

00:01:25.35 >> Of course. Do you want to start off just by telling us a little bit about yourself and your role?

00:01:30.54 >> Sure. As you mentioned, I'm the assistant director of Monitoring & Compliance, so I primarily oversee the monitoring of the indicators and then other monitoring that we do internally with LEAs, and then of course that includes the LEA determinations that we make annually, and so prior to that, I was a local director and prior to that was a special educator in the field, so I was definitely spent many, many years on the user end of this process, and so it's really exciting to have now come into the state side of it and seen some ways we can make improvements.

00:02:12.29 >> Great. I'm sure having that experience more in the field has really benefited you too to have that perspective and bringing it to the state level.

00:02:19.68 >> Yes, always think that's an important perspective to keep when we're doing this is the user end perspective, so it was a big help.

00:02:28.19 >> Yeah, especially with determinations because you used to kind of be on the end of receiving them, right, and now you're managing how they get made and sending those notifications out and that whole side of things, so very different experience, I would imagine.

00:02:41.76 >> Yes, definitely.

00:02:43.18 >> So do you want to maybe start off by telling us kind of how you are making determinations in Indiana and why you've decided to kind of do things the way you have, changes maybe that you've made to the process over time?

00:02:56.60 >> Yes, definitely, so we'll kind of start at the beginning. When I came into the role as assistant director, some of the priorities that the state had set out were around technology modernization or IT modernization, and so there was a lot of work being done in other areas of data collection for all LEAs, and so it was really important for us and our team that we were part of that. We had a system that involved a lot of Excel spreadsheets and documents back and forth and things getting lost in e-mail, and had a lot of tracking issues from things being sent to different e-mail addresses. And so we knew we had some areas that we wanted to focus on in our determination, but really one of the big push for this work was that we wanted to have special education be included in the same place that other data was going. So if data managers were submitting things, and schools were accessing their data within what in Indiana is called the LINK Initiative or the LINK Portal. We wanted special education to be there as well because I think overall that sends the message of inclusion, that we want schools to be inclusive, and if we're in a separate place, and they're searching for special education data separately, we really felt strongly that that wasn't the message we wanted to send, so that was part of the big push that we wanted to move along with that modernization initiatives that were happening.

00:04:34.65 >> Mm-hmm.

00:04:35.55 >> So with that then, not necessarily specifically for that same reason, but we started working with the National Center for Systemic Improvement, NCSI, and we were working through a process with them called leveraging our general supervision system.

00:04:53.43 >> Mm-hmm.

00:04:54.04 >> And so we started to really take a deep dive into the mapping of our procedures, and then that conversation also included our priorities, and really what were we saying to LEAs is important.

00:05:11.80 >> Mm-hmm.

00:05:12.11 >> And that definitely needs to come across in our annual determination. So that led to again not just the process of how you receive it and how you access the data to what was included in that determination process, and what information were they looking at?

00:05:31.47 >> Right.

00:05:31.71 >> So with that process, we did, we brought together a fairly large stakeholder group that was hopefully representative of not just LEAs but other organizations that we work with and again started to look at our priorities. What's our vision? What are we currently doing that speaks to that vision? And then maybe what things were being included that we were not hearing at this point were as important as maybe they were when that process was first developed.

00:06:05.43 >> Mm-hmm.

00:06:05.72 So we ended up removing several things, so for example, we heard loud and clear, graduation rate was important, and we wanted to keep that as a focus area.

00:06:20.64 >> Mm-hmm.

00:06:20.98 >> So one of the things we did was increase the weighting for the determination calculation related to graduation rate, but then one of the things we also heard was that our preschool outcomes data was not exactly where we wanted it to be related to our assessment outcomes and how schools were able to use that, and so we removed that component right now until we can get that fleshed out and reworked kind of what then was included and how things were weighted based on that stakeholder feedback and engagement.

00:06:55.97 >> Okay. Were there other elements that you added in that maybe you didn't have before kind of based on the stakeholder engagement or just the priorities of the department?

00:07:05.73 >> Yeah. At this point, we did not add as much as we changed weighting.

00:07:11.53 >> Hmm.

00:07:11.74 >> So we were very heavily also focused on least restrictive environment and an inclusion of school-age children, and so we had done really well, and we were at the higher ends of performance there, and again recognizing that we don't want to say that inclusion is the only placement, and recognizing that there is a continuum of placement. We reworked that weighting because we still want to say, "This is important, and it's a focus area, but we're doing well, so let's acknowledge that schools are typically doing well, and then focus more on" ... And where we shifted that was to growth, so our assessment growth was not weighted as highly as some of the other components, and so we wanted to say, "We're not so worried about your proficiency level as we want to focus on paying attention to growth of students in terms of the assessment versus being proficient," so while those things were included, we kind of de-emphasized and re-emphasized certain points there.

00:08:22.75 >> Mm-hmm. Yeah, that makes sense.

00:08:24.58 >> So from there, we were able to develop many more resources because we wanted to release this at the next annual determination time. We wanted people to really understand what it meant, where the data came from, and then resources. That if this becomes an area that you see from your determination you want to address, you've got in one place all of that information. And so for each of the items we included, we reworked some of our compliance and monitoring guides and included the determination information in there, again speaking to, how do you locally find this data? What report does it come from, and when is that report collected? When do we make those determination, that whole timeline process specific to what we call our results-driven accountability, which is our annual determination. And so along with that then, we also, since this was a new system, we had to develop resources about just accessing the system and things like, what role do you have to be assigned to in your district because it is a secure portal?

00:09:39.89 >> Yeah.

00:09:40.13 >> What roles do you have to have to access the data? Who can access the data? How do you and the district ... Who do you want to access the data and at what level? All of those kind of things had to be developed along with the actual development of the system. So we definitely wanted to be transparent, and we wanted to have resources available that people understood how to sue the information, and not that there weren't things before but just really, again, as I've said, re-emphasizing the importance of these things as we were kind of doing a grand reopening sort of of this work.

00:10:17.93 >> Mm-hmm.

00:10:18.17 >> So that also took some time, and all then was posted to our website prior to the opening of this dashboard.

00:10:27.82 >> And when did it first open?

00:10:30.36 >> So our annual determination is ... happens annually in November, so November of last year was our first release, and it surprisingly went really, really well. We had very few bugs. We did have a few but had very few, but just definitely really well received by the field. We had a lot of feedback that all of the things that were posted within the dashboard were also able to be downloaded into an Excel spreadsheet so directors who had access could then easily download the information to share with their staff, but they had the primary access to that information. So from a technology standpoint, things went really well.

00:11:21.00 >> Great.

00:11:21.17 >> And then the feedback from the field was really positive, and they appreciated I think also just having things in one place, that they weren't going somewhere else to get that information, and then always available to them. It's still there. We have last year's information. We on the back end are loading this year's data, but they'll be able to then continue and have that data longitudinally. We're also looking at loading previous or historical data. We haven't gotten to all of that, but we hope in some of our next iterations to be able to do some trend analysis for the districts as well.

00:12:03.94 >> Wow, it sounds like a really robust and useful system, so exciting. Are there other benefits that you've seen to this new system?

00:12:14.76 >> Yeah, I think it's also helped because the way we set it up is different pages, so previously when the LEAs received this determination, it was kind of one long running page.

00:12:26.51 >> Yep.

00:12:26.68 >> It'd be three or four pages long, but it ran, but in this system now when the staff open up their portal, their dashboard in the portal, they will see different pages.

00:12:38.05 >> Mm-hmm.

00:12:38.31 >> So the first page is actually the discrimination and kind of the summary, so they started off, they see where are we from our determination and our level of support ratings to them breaking that down, so we have our compliance page, which then speaks to their determination, and so that made it much clearer, I think, how exactly that determination is calculated. Then within the system, each of those also, if it was applicable, we had a little information button so if they wanted more information immediately about what that particular indicator was, they could open that up, and it would give them a little bit more information kind of in the moment.

00:13:23.85 >> Mm-hmm.

00:13:24.43 >> Then another page then was the results page, and so then we had our indicators that speak to results listed there.

00:13:33.33 >> Mm-hmm.

00:13:33.76 >> Then we had ... were able to also add what we called a non-scored data page, so it was data that we wanted to share, but it was not calculated into the determination, but it's things we want to keep on the radar and we want to keep having conversation about. So one example is our presence LRE was not included as a scored item, but we want to have a lot of conversation about that, and it may be in the future, but we want to get people to start looking at that data as well and know again that we think it's important, and so we want it to be included in some way. There were other indicators that weren't scored and included, but it was a way for us to acknowledge that, "Hey, we're looking at this. Hey, we want you to pay attention to this." And then there was another page that kind of summarized and gave some next steps, so one of the things we were also able to do was incorporate some of our corrective action requirements into the system, so it was kind of everything in one place. So we were able to set up some Jotforms where they were linked in that final kind of summary ... or instructions I think is more of a better way to say it ... page that if you had a finding related to initial evaluations, there was a Jotform there where we would start that corrective process there within that system, and then had it set up where then we could get that data directly. Again, it wasn't e-mailing things back and forth. It was in one spot, and on the back side then that they gave us another great way to collect data more specifically into the noncompliance of the individual LEAs to help them help and support and kind of see from a state level, again, "What resources are we needing to develop?" because of some of the analytics we could do with that data collection because prior, although we were collecting it and tracking it, it was kind of in so many different places, it was hard to make sure we were getting things directly. It might have to go through two or three different transitions to get to the right place, so think of kind of a one-stop shop that you're doing related to your determination and correction was all there.

00:15:56.61 >> Yeah, that's wonderful, and I think, yeah, that one-stop shop and having everything in one place makes such a huge difference I'm sure for you all at the state and then also the districts too, and probably has really helped to improve the quality and timeliness and accuracy of all of the data, I would imagine, instead of having things in different places and having to track things in different ways that can kind of lead to more of those data quality issues. If it's all in one place, it seems like it would be easier to ensure the quality.

00:16:30.47 >> Yes, you described that perfectly and really what our goal was, and then fortunately that's really what we saw comparatively to how we had been managing that before.

00:16:42.77 >> It sounds like you had a really good launch of the system. Were there any challenges that you have encountered though, and how have you addressed them?

00:16:53.32 >> Yeah, it's been a lot of collaboration. We did work with an external vendor in developing the actual dashboard itself. It is run by Power BI, and so we have closed out with that vendor, so this next round is where we are working internally with our IT department, and so this is again not right at our fingertips. We have to go through another department to make sure that data is loaded, and then let ... be loaded in time that we can do some quality checks on that, so while it's our project, and we're leading that project, so having to reach out and have communications with other departments that have lots of projects going on, our IT department is working with everyone really, so making sure that we're getting on their radar and staying on their radar because we have data coming in all the time, and so that will be ... We're in that process, and although right now it's going well, sometimes it's not as fast as we would like to see it go because we don't have the direct access to loading the data.

00:18:07.89 >> Yeah, that all makes sense and I'm sure is something very common in a lot of the work that you do in other states as well. It's kind of when you have a vendor, there's things that you're working through, but then similarly you would think, "Oh, if we bring it in-house, it'll be different. It may be easier, and we'll have more control," but then there are other hoops you have to jump through, of course, doing it that way, too. So that definitely makes sense, and so far that's great that it's been a pretty smooth launch and new system.

00:18:38.76 >> Yeah, we're definitely hopeful for the next round. Again, you don't know what will come up, but we're definitely trying to get ahead of things, get things loaded and reviewed as soon as possible as far as is reasonable, given when times for data collections and things like that are happening, but we're feeling good. Our IT department is really confident that things will go well, and so again given the success we had last year, I think it's also reassuring that people will ... this year, we won't have the new factor.

00:19:15.97 >> Mm-hmm.

00:19:16.47 >> Know where to go, they'll know how to log in, so a lot of those things that we struggled with last year, we're going into this year thinking we're not going to have as much of that kind of questioning or issues.

00:19:27.37 >> Mm-hmm. Yeah, there's always a learning curve, so hopefully, yeah, this year will be even easier. Do you have plans for the future in terms of maybe other changes or what you hope to do moving forward with the system and determinations?

00:19:42.42 >> Yes, definitely. I mentioned we want to load previous data, and we're having some discussions about that, but also then as we're moving forward, one of the main things we've had discussion around is, right now this is district-level data, and so we want to move to school and student-level data. Although it's the school's data, what we find especially for our special education directors is that they aren't always the ones that have access to that data directly, and so we feel like if this is something we could provide for them, it just helps them be able to better use their data, so the assessment data, while that comes in and the school has that, sometimes that's ... in our state, the testing coordinator would have that data and not necessarily the special education director, and so helping to drill down for LEAs that may not have that available to them directly, that is something we've heard back would be a next step, potential next step.

00:20:59.30 >> Great. Is there any ... I know this is all not information that the public can access since it's the district's data, but is there anything on your website that if other states are interested in just getting a sense or a taste of what this is like, there's something that they might be able to look at? I don't know if you have screenshots or ...

00:21:21.09 >> We do, yes, yeah.

00:21:22.84 >> Great.

00:21:23.15 >> So on our website, we do have a tab that is called Results-Driven Accountability, so that specifically, we wanted it direct so that the field could get to those resources directly, and there is a user guide at that location, and so that, like you said, it has screenshots. It has actual information about what is included, and then there are other resources that clearly define how we're weighting those measures. What weight ... what measures are included? How are we weighting that? Exactly how is that calculation met? There are certain things that ... our score, we use typically a one through five scoring, and so how we assign those scores to different districts. We do use an enrollment size. We have such a wide variety of schools in our state from very small rural schools to large urban schools, so we have some different calculations that we use based on your enrollment size. So we're hoping that measures apples to apples ...

00:22:34.20 >> Right.

00:22:35.32 >> ... because what we were hearing from people is that we're just so different that we really shouldn't be compared to each other in certain ways, so that was something that really originally in the original determination process was determined, but we have continued that.

00:22:55.11 >> Okay. Well, we'll definitely put a link in the description of the podcast episode for anyone listening that wants to check out and see more of what Angela's been describing to us. I'm sure people will.

00:23:07.61 >> That would be great, and we'd be happy to answer any questions or share any more information that we can about what we went through.

00:23:16.12 >> Great, well, thank you so much for talking about your amazing, robust process. We really, really appreciate it, and thank you for being on.

00:23:27.27 >> Well, happy to be here and definitely appreciate the opportunity.

00:23:32.89 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.62 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.45 >> Hey. It's Amy, and I am so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.60 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.94 >> Welcome to part two of my conversation with Leah Voorhies, who is the Assistant Superintendent of Student Support with the Utah State Board of Education, and Alissa Trollinger, who is the Deputy Associate Superintendent for Exceptional Student Services with the Arizona Department of Education. We are going to pick up where we left off last time as they continue to share their states' stories about how data play an important role in their initiatives and priorities.

00:00:56.23 We've talked about a number of things I think that are really exciting and interesting. What's something else that you're really excited about that's going on right now?

00:01:06.54 >> Well, we have finally, I would say within the last 6 months, been able to get some traction with our School Improvement. I'm really envious of Leah's ... I'm not envious. Let me take that back. I was going to say I'm envious of Leah's oversight of School Improvement. I'm not. I don't want to do that. But I am envious of the relationship that she's been able to create because I think we've really struggled with that in the time that I've been state director. And we have a new superintendent of public instruction who started in January, and there's been some change at the agency leadership level, and with that we have been able to start working with School Improvement to try and work across systems to make change happen. So that's really exciting. I think that gives us some opportunity to scale up structures that we already have in place to do root cause analysis and to support improvement in literacy achievement in particular. Our state systematic improvement plan is geared around K-3 literacy achievement, and it's already aligned to our state's Move On When Reading initiative, which basically is, once any child has hit third grade, we want them to be able to be literate. Aligning our asset literacy goal with the Move On When Reading initiative has meant that we have access to a lot more benchmark assessment data than we had previously. In Arizona, it's not a requirement that schools submit periodic updates on their data ...

00:02:52.76 >> Mm-hmm.

00:02:53.09 >> ... on their children's achievement, and so we've been able to leverage the Move On When Reading requirements in order to help us understand what's happening with our root cause analyses. We've tied our SSIP goals to our programmatic monitoring system, so again, that's a way for us leverage participation and to ensure that we're able to do deep dives and that the LEAs are committed to this work. But now I think with the relationship that we're forming with School Improvement, we might be able to scale up some of these efforts. I think one of the challenges for us is to do exactly what Utah has already started, is to really dive into the data and look at it at school level, so I think that that's one of the things that we're going to be building into our dashboards.

00:03:42.10 >> Great, and you have Leah to turn to for ...

00:03:45.04 >> I do.

00:03:46.65 >> ... what they've been doing and learn from them. Leah, what about you in Utah? What's something going on that you're really excited about?

00:03:56.57 >> The work that we are doing across School Improvement and Special Education is really exciting to me. This year, our team ... so an individual from the Special Ed team and an individual from the School Improvement team coached five schools. After we did the regression analysis of schools that were including students more than the state average and that had better or worse outcomes, we then factored in the schools in the state who had been identified for targeted support and improvement only for students with disabilities. So if they had multiple student groups, we pulled them out of the calculation ...

00:04:47.92 >> Mm-hmm.

00:04:48.14 >> And so we only ended up with seven schools, oh, except that we also added in the LEAs who had high risk in our results-driven accountability matrix. So the LEA had to have high risk. The school had to be identified for TSI only for students with disabilities, and they had to have better than ... They had to be including students with disabilities at least at the state average ...

00:05:13.17 >> Mm-hmm.

00:05:14.48 >> And that only yielded seven schools across the state, which is fascinating. We have almost 1,200 schools. So we contacted every one of them, and we said, "Hey, we'd like to help you improve outcomes for students with disabilities, which will help you exit TSI status, and we'd like to help you do that compliantly so you help your LEA improve your results-driven accountability risk related to compliance and related to indicator three." And so five of them took us up on it, and we have been coaching five schools across the state this year, and the major focus has been having them look at their data from different perspectives, having them look at any and all program improvement plans, school improvement plans that they have because they have multiple at the LEA and/or the school level, creating one comprehensive plan with prioritized action steps, 90-day action steps, and then bringing to bear all of the resources at the state level that we can to be able to support them, professional learning, technical assistance, special grants sometime if they need it for personnel, that kind of thing, and it's really exciting. We have ... We are forming really good relationships, and we have other schools that are interested in participating this year, and we're looking forward to the School Improvement results that will be coming out in the fall to see if these schools are making enough progress that they would be able to exit next year or ... They get 4 years, but it would be great if they could exit sooner rather than later.

00:07:15.32 >> Yeah, and then you could, yeah, see those kind of hopefully earlier successes, too ...

00:07:20.50 >> Yeah.

00:07:20.69 >> ... and be able to start ...

00:07:21.43 >> Hopefully there's a trend, yeah ...

00:07:22.99 >> Yes!

00:07:23.68 >> ... that it's making a difference, and then if other schools are interested, our ... The current work that we're doing internally is, how can we scale this up?

00:07:31.84 >> Mm-hmm. Mm-hmm.

00:07:32.47 >> What is our capacity to scale it up?

00:07:35.28 >> And this is beyond Special Education. This is ...

00:07:38.40 >> Right, yeah.

00:07:38.91 >> ... across the ...

00:07:40.03 >> Yeah.

00:07:40.13 >> Wow.

00:07:40.31 >> We're calling it Integrated Coaching because it's School Improvement and Special Ed, are integrating the practices of the two different sections of our agency.

00:07:54.88 >> Yeah. Wow! I've been hearing, I feel like, more and more states are sort of trying to go more in that direction of not having it also siloed, but if there are struggling schools and districts, and there's all these different requirements for corrective action and those sorts of things, it really should kind of not be done in isolation. It should be because all of these issue and challenges and things going on are all intertwined. So it just makes sense to kind of bring everyone together to try to tackle those issues. So that's really neat to hear about.

00:08:31.84 >> It does, and it's been really interesting to look at the leadership level of the LEAs because when we started, we ... The School Improvement director and I, we contacted the LEA Special Education director and the Title I director and said, "Hey, we have this idea." And then they all contacted at least an assistant superintendent.

00:08:53.35 >> Mm-hmm.

00:08:53.60 >> In one of the five, we have the superintendent that participates in the meetings. So there's one of the side benefits, not a side effect, but a side benefit, is that the title office in the LEAs is really learning about results-driven accountability and learning about Special Education compliance and learning about the SPP and the APR, and the Special Education director ... Although we have tried to provide technical assistance to Special Education directors about every year about the schools that are identified for TSI ...

00:09:32.99 >> Mm-hmm.

00:09:33.23 >> Every single year we provide them information. They don't really get it. But the ones that we're working with now are getting it because they're really diving into, how was this school identified for TSI, and what do they have to do to exit?

00:09:49.48 >> Mm-hmm.

00:09:50.25 >> So that's been a really good benefit. Of course, it's a very, very small sample size, but we hope we'll increase it.

00:09:57.68 >> Yeah.

00:09:58.50 >> Well, and, Amy, I'll say this is really exciting to listen to, and, side note, this is one of the nice things about working with really, really talented State Special Education Directors as colleagues because you learn a lot, and you take a lot of notes from people who are doing good work. That's exciting to hear. Everything you've just described, Leah, is exactly where we want to go. I think the struggle for us is helping all of the parties at the leadership level, both internally and externally, understand that Special Education isn't the job of Special Education alone because we keep having to remind folks that our students are general education students first and that these efforts to support with professional development and technical assistance need to come ... We need to come together to provide that. There's a tendency internally, I think, and externally to shuffle off that work to Special Education only, and so I feel like we're really trying to help model that at the agency level so that we can get that work at the local level done properly. And the fact that you're overseeing that really, really helps, I think, manage resources and ensure that the work is done in an integrated way, so that's very, very interesting and hopeful for me, that maybe that's a place we can get to.

00:11:22.84 >> Yeah.

00:11:23.11 >> And the hope for me is that it actually makes a difference.

00:11:26.61 >> Sure, sure.

00:11:27.71 >> And then having those stories and being able to share those out within the state but then also talking to your districts, too, in Arizona and showing them, "Look at what this kind of collaboration, how this really helped improve results," will kind of be the carrot.

00:11:47.34 >> I know. It's exciting. We expect big things from Utah so that we can turn around and do it.

00:11:52.14 >> Yeah. Uh-huh.

00:11:52.81 >> They can lead the way and then ...

00:11:54.25 >> I do, too!

00:11:56.76 >> So what about plans for the future? What do you have coming up? Alissa, do you want to start that off?

00:12:03.29 >> Sure. Again, our big plan at this moment, I actually have a meeting today to figure it out, is to redesign or reorganize our internal structure to be ... to provide professional development and technical assistance in a more data-responsive way. I feel like if we can set our internal structure in leadership and personnel and data practices, then we're not going to be ... We're trying to get out of a spray-and-pray model where we hope that we're hitting the right marks. We actually want to use our data, plan our ... and plan our resources efficiently and in a targeted way. And I'm sure that that may sound simple to people who are listening, but it really does take a lot of effort, and in order to even get to this place, we've had to make sure that our data is right.

00:13:03.39 >> Mm-hmm.

00:13:04.00 >> And so that's really exciting work, and it's kind of challenging work because it means we are asking people to rethink the way that they've offered professional development. We may be cutting contracts with long-standing projects that have meant a lot to us over time, but if the data shows that they're not making an impact, we need to move forward with things that will make an impact.

00:13:29.07 >> Yeah.

00:13:30.23 >> So that's a lot of the internal work that we're hoping will help us focus our efforts on increasing achievement, and that's going to be a lot for us over the next couple of years.

00:13:43.31 >> Mm-hmm. Yeah, that's a big chunk to bite off, but ultimately, yeah, for the best and really being able to use that data at all levels for the improvement.

00:13:56.55 >> Yes.

00:13:56.92 >> Great.

00:13:57.09 >> Yeah, that's really courageous work, so as long as I'm here, Alissa, I'm your cheerleader.

00:14:04.74 >> Thank you so much, Leah.

00:14:07.55 >> In Utah, we are ... We're really looking at ... internally at making sure that as EDPass is implemented that we don't have the timeliness and lack-of-accuracy issues that we have had in the past. And it's not consistent issues, so it's not like we can say, "Here's the thing that we've done wrong." I think a lot of it has been related to personnel changes, and we have a consistent team now, and so we're hoping that that will make a difference for the next couple of years. So internally that's one of our big goals. Externally, we are ... We will be expanding the integrated coaching that we are doing. So I will be reporting to the schools that, in year two ... I just got the data recently from our statistician about which schools are doing better than expected and which schools are not doing as well as we would expect, and I'll start meeting with leadership teams here in the next couple weeks to talk to principals and leadership teams, Special Ed directors, Title directors about how we, at State, can support improvement when it's needed, and those teams that are doing better than expected, what they're doing that we can add to our list of promising practices that we are then going to be very systematically sharing with the rest of our state. And so we're putting together the dissemination tools, the systematic dissemination tools, to be able to advertise those bright spots across our state, those promising practices, and to be able to provide the tools that other schools need, putting together a systematic dissemination plan so that any school in our state who is interested, whether or not they are doing well or they need to improve, because students with disabilities, proficiency is low everywhere. Even for the schools that are doing better than expected, it's still not good.

00:16:36.10 >> Yeah.

00:16:36.71 >> So every school in the state would be able to access information about how to implement the practices that we have found are working across our state, and so that's our next step. And in doing that, we have to have the data available and in a format that anyone in the school can make sense of it and can go, "Oh, this is where my class, my students, the students that I provide services for, the students that I'm responsible for, fit into this story, and this is what I can do to work on improving outcomes."

00:17:23.74 >> Mm-hmm. Yeah, really making it so that it's understandable to everybody and that they'll ... It's great to see things that are working and what's going well, but taking that next step to really then be able to connect it to themselves personally and what their context is ...

00:17:42.78 >> Exactly.

00:17:43.32 >> ... is a whole other, yeah, thing. So that's ...

00:17:46.61 >> And it's interesting, I'll just state. So I've ... Like I said, I've been going around the state talking to people about their data, and I tell them, "You gave us this data. Then we analyzed it. Now I'm giving it back to you," and it's interesting to me how many people say they don't believe it.

00:18:04.40 >> Yeah.

00:18:04.56 >> Yes.

00:18:05.33 >> So many people say, "Oh, I don't believe that." I'm like, "Oh. Well, you gave it to me."

00:18:12.30 >> Or they make an assumption that we've done something to it.

00:18:15.45 >> Something to it, yeah.

00:18:16.16 >> Mm-hmm. Yeah.

00:18:17.84 >> I would agree.

00:18:18.97 >> Yeah. It makes you feel like, well, they're not necessarily looking at their data themselves before it gets submitted, and that might be an important step and help convince them that, "This is really the data that you gave us." So, yeah, that is something ...

00:18:34.30 [ Chatter ]

00:18:34.30 >> And also I don't think they want to take responsibility for the fact that students with disabilities are not doing well.

00:18:40.30 >> Yeah.

00:18:41.29 >> Even the ones that are doing better than others are not doing well. There's a huge gap, and everyone is responsible for that gap.

00:18:50.60 >> Yep. Yeah. They don't want to believe it really is true.

00:18:53.99 >> I would have to agree with Leah on that, as we've really over the last couple of years done the ... I call it the SPP/APR data visualization with people, the data conversations. I think it's been shocking for stakeholders, parents and schools to understand how poorly students with disabilities are doing. And I think that the struggle always is in helping LEA general education and administrators understand that students with disabilities are gen ed students first and that they need to invested in in ensuring that they get instruction and services just like any other child. Even internally I've heard, "What is Special Education going to do about that?" And my response back when I hear that is, "What are we going to do about that? Because they're also you're kids."

00:19:53.29 >> Yeah, agreed.

00:19:55.97 >> Well, it sounds like there's a lot going on and a lot you have planned, and this has been such an interesting conversation. I've loved hearing all about your stories and the great work you've been doing, so thank you both so much for joining.

00:20:11.44 >> Thank you for having us.

00:20:13.02 >> Thank you. This has been fun.

00:20:15.35 >> It has been.

00:20:17.32 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.62 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.60 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.85 >> Hello. On this episode of "A Date with Data," I am joined by the directors of special education from two neighboring states. We have Leah Voorhies, who is the Assistant Superintendent of Student Support with the Utah State Board of Education, and Alissa Trollinger, who is the Deputy Associate Superintendent for Exceptional Student Services with the Arizona Department of Education. And they will be reflecting on their states' IDEA data quality and really looking forward to hearing what each of you are focusing on and excited about and maybe if there are similarities between kind of what you're doing but also really curious about your unique journeys as well. So welcome to both of you, and ...

00:01:07.32 >> Thank you for having us.

00:01:08.42 >> Thank you.

00:01:08.67 >> Yes. So can you each tell us a little bit about yourselves? And, Alissa, if you want to start off.

00:01:14.99 >> Sure. Thank you. I am in my eighth year as State Director of Special Education in Arizona. But this is my 18th year with the state in our Special Education Department. And it's been quite an adventure in my role, but I feel like we are really privileged to do a lot of important work.

00:01:42.05 >> Thanks. Looking forward to hearing more about your adventure, and Leah?

00:01:47.11 >> I've been the State Director for this going on 7 years. Prior to that, I was a coordinator, which is like the assistant director ...

00:01:58.46 >> Mm-hmm.

00:01:58.73 >> ... for another 6. So it's been 13 years. We have a very robust team in Utah, and we have several important priorities, and data play a huge role in each of those priorities, and we're working on improving meaningful, making inclusion more meaningful for students with disabilities and improving postsecondary outcomes and improving the number and the ... So the preparation and the recruitment and retention ...

00:02:35.83 >> Mm-hmm.

00:02:36.25 >> ... of both paraprofessionals and professionals in special education, and all of those initiatives require an enormous amount of data analysis.

00:02:46.51 >> Yes, I would imagine so. Okay. Well, that was a little taste of some of your priorities. Leah, do you want to talk a little bit more in detail about some of your IDEA data quality priorities in Utah?

00:03:01.44 >> Sure, so for the past decade, we have been having what we call data literacy deep-dive conversations with our LEAs about all of the IDEA indicator, SPP/APR indicator data. So we spend a week traveling around the state and meeting with leadership teams from all the districts and charter schools in Utah, which right now is about 160-ish, looking at all of their indicators and asking them questions about their data, about the quality and about what it actually means and what they might want to do about it. And then we generally help them create program improvement plans. And as they've gotten better and better at that, they've asked more in-depth questions and the past 2 years, our special education advisory panel has also asked more in-depth questions because we reviewed the APR/SPP data with them every year as well.

00:04:12.21 >> Mm-hmm.

00:04:12.52 >> And they've asked specifically about the relationship between the amount of time students with disabilities spend in general education and their proficiency and their graduation and their drop-out rates and the post-school outcomes. So one of our major data initiatives is, we have an external statistician who is doing regression analyses for us, just dumping in all of the data that we have about outcomes and about least-restrictive environment and doing it by both LEA and by school. And then we are working with our school improvement team, which I also supervise, the Federal School Improvement, but we also have a State School Improvement team to compare their data with special education data, and we're working with schools across the state to specifically look at school improvement efforts for schools that are identified for targeted support and improvement for students with disabilities, so that's a major initiative that we have going ...

00:05:14.77 >> Uh-huh.

00:05:15.32 >> ... that has come out of, as a result of LEAs and the Special Education Advisory Panel asking questions about our SPP/APR data.

00:05:25.55 >> That is very cool. And have you gotten results yet from the statistician and started to use those data and analyses?

00:05:34.81 >> We have. So I've been meeting with the principals and special ed directors across our state who have both better-than-expected results, not average, but expected results ...

00:05:51.03 >> Mm-hmm.

00:05:51.44 >> ... and not-quite-as-good-as-expected results. I haven't been meeting with principals about that. I've been meeting with special ed directors. But I've been meeting with principals and leadership teams across our state, talking about their, having them tell me their story. What is it that you are doing that you believe is contributing to your better-than-expected results? And it's so fun, and then we're reporting those and aggregating the information about those stories, and we're reporting them to all of the special ed directors in the state and anyone else who will listen.

00:06:31.75 >> So, Alissa, what about you? What's happening in Arizona in terms of your data priorities?

00:06:38.34 >> Well, I think that Leah and Utah are a lot further along than we are. But I'm pretty proud of the progress that Arizona has made to structure our data systems and begin to build our data literacy culture. I'll kind of give a beginning so that it's more understandable about why we're in this space and why we were kind of on an adventure. In the 18 years that I've been here, before I was the State Director, we had 12 State Directors ... excuse me, five State Directors in a 12-year period, and each of those State Directors had varying degrees of understanding and experience and comfort with data. And I feel like when I started in this position, and my leadership team started, we were really beginning from scratch to build our entire, really our entire system of general supervision, putting it back together. Some pieces had existed previously and dismantled, and some pieces didn't exist at all. And so we've spent really the first part of my time in this position putting systems back together, making sure that we have programmatic monitoring system that functions correctly, making sure that we are able to collect data accurately and with validity. And that's kind of a challenge in Arizona because we have almost 700 LEAs.

00:08:10.11 >> Wow.

00:08:10.39 >> And about two-thirds of them are in charter schools ...

00:08:13.16 >> Oh, my god.

00:08:13.49 >> ... charter schools. Mm-hmm. Charter schools in Arizona stand as their own LEAs, and so they're each responsible for providing special education and related services but also submitting their own data.

00:08:26.78 >> Mm-hmm.

00:08:27.20 >> And so if you think about our data-collection system, not only did we need to have a structure that was built to collect all of that data from all of those differing LEAs, but we also needed to put a system or put together a system where we could continually train on how to do this because we have a high turnover ...

00:08:49.85 >> Mm-hmm.

00:08:50.16 >> ... in data staff at LEAs, particularly in charter schools. So that's really what we've invested a lot of time in. We've also worked really heavily in data governance in our state to make sure that the decisions that are made at the leadership level within the agency reflect accurate practices and that other parts of the agency understand special education data reporting requirements. I don't know, Leah, if you experienced this, but we often, I'm going to say with regularity, have to explain to the rest of the agency that there aren't any do-overs in the IDEA. Our data has to be collected and submitted timely and accurately and it, to do other than that incurs a penalty. And that's been something that the rest of our agency has really had to kind of get on board with. We need to all be on the same page. Our data terms need to be consistent. Our timelines need to be consistent. We all need to be working together to help support IDEA data collection. I think that's what everybody's goal is, but it puts an extra level of urgency in when we don't get any do-overs.

00:10:00.72 >> Yeah, especially now ...

00:10:01.79 >> So ...

00:10:01.91 >> ... with EDPass and modernization ...

00:10:04.20 >> Exactly.

00:10:05.10 >> ... really, yeah. The first time, it's got to be perfect, pretty much.

00:10:09.75 >> I think that Alissa is being very humble because I've been the state director the whole time that she has, so we're good friends.

00:10:17.94 >> Mm-hmm.

00:10:19.20 >> And the amount of systems work that Alissa has done, because we talk at least monthly. The amount of systems work that Alissa has done, it is really amazing, especially when you really think about 700 ...

00:10:37.66 >> Right.

00:10:38.04 >> ... LEAs.

00:10:39.62 >> Yeah, and the majority of them are single ...

00:10:43.11 >> Single buildings ...

00:10:44.28 >> ... schools.

00:10:44.62 >> ... absolutely.

00:10:45.25 >> Yes ...

00:10:45.48 >> Mm-hmm.

00:10:45.90 >> ... small LEAs, yeah.

00:10:47.51 >> Yeah.

00:10:48.79 >> And I completely agree with Alissa. I have been super frustrated with some of the processes in our agency to submit data, EDFacts data.

00:11:03.65 >> Mm-hmm.

00:11:04.04 >> And the lack of understanding, and sometimes it seems like the lack of caring, but it isn't. It's a lack of understanding ...

00:11:12.57 >> Yes.

00:11:13.29 >> ... about the high stakes ...

00:11:15.15 >> Mm-hmm.

00:11:15.46 >> ... of IDEA data. Two years in a row now, I've had to sit the Deputy Superintendent down with the Director of Data and Statistics and say, "Look. Your staff did not do their jobs in an accurate and timely manner, and this is the consequence for my staff and for the state." I don't think it had ever registered that IDEA data can, is really high-stakes, and the difference between getting a determination of meets requirements and a determination of needs assistance is really upsetting to people.

00:11:54.96 >> Yep.

00:11:55.40 >> Yeah.

00:11:56.62 >> And my internal team didn't get that, and I even invited them. I sat down, made them come to a discussion with OSEP data people so that they could hear it from someone besides me. And Alissa and I have talked about this a lot, so I know how many times she has done that, as well.

00:12:15.53 >> Absolutely. That has been really helpful, to have OSEP's support in articulating the rigid data reporting requirements to help the rest of our agency understand. And it's funny when I reflect on it. Years ago, when I supported data submission, we were having the very same conversations that we have now ...

00:12:35.32 >> Mm-hmm.

00:12:35.65 >> ... in terms of ensuring that all of the agencies onboard that were working on accuracy, but it's not only training our external stakeholders but ensuring our internal stakeholders are on the same page.

00:12:51.12 >> Agreed.

00:12:51.60 >> So in terms of our priorities, our priority, I think, at this point in Arizona is to make sure that we are not only increasing our data literacy but using all of the data that we believe is accurate and valid at this point to make good decisions. And so really we've been engaged in restructuring our priorities based on data, putting together data or, actually, we have some data dashboards, but enhancing those data dashboards and getting information out to LEAs so that they can make timely and meaningful decisions with that data and just really focusing our efforts on improving outcomes. It was really hard to get a grip on that when we didn't have confidence in our data.

00:13:40.15 >> Yeah.

00:13:40.27 >> But now that we do, and we have system that are working well, that's our challenge is, to make sure that when our special education advisory panel or our parent stakeholders ask questions, that we are able to dig into the root cause and get some answers.

00:13:58.31 >> Wow, great. Well, yeah, it sounds like for both of you, feeling at this point pretty good about the quality of your data, and that goal is always to get there and then be able to use it for improvement and decision-making, and it sounds like you're well on your way. So that is very exciting. And also, you kind of touched already a bit on building that data culture within your state, whether that's within your special education department but also really beyond and making sure that everyone throughout the SAA really understands the importance and what this all means and why having high-quality data is so critical. Leah, do you want to say anything else about kind of how you've been building that data culture in your state?

00:14:44.32 >> I think it has made a huge difference for us to take the time to sit with LEA special education leadership teams and talk through, and we provide thought-provoking questions that they can ask, that we can ask them, that they can ask us about their data and talk through, like Alissa was saying, their system for collecting and for verifying and for submitting their data because so many LEAs have such different systems and so not necessarily just hearing from us. We'll get them the information that they need to feel confident in their processes because they need it to be very LEA-specific.

00:15:36.36 >> Mm-hmm.

00:15:37.02 >> So going to sit with them or doing it online, doing it virtually. And we make it a point to let them know that we are willing to come to them. We don't make them come to us. They can meet with us virtually, or I'll send a team. And hopefully that helps them understand how important it is. And I do think it's made the difference. We have some LEAs that are getting pretty savvy about their data now and can speak very confidently about their data. And like Alissa, we have over 120 charter schools, so we have more than two-thirds of our LEAs that are charter schools, and some have multiple buildings, but most of them are one-building charter schools. And there's an enormous amount of turnover. So even though we have LEA leadership teams and special ed directors that are really savvy, we also constantly have new leadership teams and new directors that have no idea where the data comes from. They don't even understand that they give it to us before we analyze it.

00:16:49.49 >> Yeah.

00:16:49.58 >> Right.

00:16:52.76 >> So we have really had to think critically and intentionally about having a tiered system of supports for LEAs because we have really savvy directors, and we have brand-new, 22-year-old, right-out-of- bachelor's-degree-program directors.

00:17:14.74 >> But it sounds like you have a robust system in place to really support them and realize that challenge of the turnover but being prepared and having those resources and technical support in place, I'm sure, is really making a big difference.

00:17:31.24 >> Yes, and the system constantly has to be tweaked because every year, we'll learn that there's a gap or that there's a great new question that we want to address. And then the processes change, too. Systems change. LEAs' student information systems change. And our system changes, and OSEP changes the calculations on us.

00:18:01.14 >> Mm-hmm.

00:18:01.35 >> Yeah.

00:18:01.69 >> Yep.

00:18:02.26 >> So it's a good thing we believe in continuous improvement because it's not an option.

00:18:08.34 >> Right. You don't have a choice.

00:18:09.98 >> It's definitely never static, and I would agree with Leah, I think, as I think about that question. In Arizona, we have really solidified our data reporting internal structures so that now that we have built accurate systems internally, we can continue to maintain them, and that has meant that we expanded our internal staff to support data collection, to centralize data collection efforts, at least to push out data collection and building strategies out to different parts of exceptional student services in the agency. And now we centralize it with a data management team, a data operation team, and we think that, that will help us maintain our structures moving forward.

00:18:57.66 >> Mm-hmm.

00:18:58.20 >> It will also be working with the IDC to document all of our data-collection procedures, so we're really excited to get moving on that work and put that documentation in place so that it will outlive us and our positions. I think we have established a data culture for a long time primarily through our programmatic monitoring system, and that is not to say that all the data comes from monitoring. But we have used our specialists who work within the programmatic monitoring system to be kind of our data ambassadors. We use a risk analysis tool that is really just a giant data dashboard that contains not only indicator data from the SPP/APR but also pieces of data from all of the data that we collect from LEAs. And every year, we are updating that information and presenting it back to LEAs using specialists who case manage those schools in our program monitoring system. So our programmatic monitoring specialists are kind of jacks of all trades. They understand what each of the data pieces are. They report that back out, and they talk about data quality, data trends. We're really trying to keep data literacy from superficial to deep understanding on the tip of LEA special education leaders' tongues, and that doesn't necessarily get into the deep root-cause analysis or trending, but it does at least help them be refreshed about what the data that they're submitting means. As Leah said, a lot of times, they don't even realize that they are submitting the data, so this helps to keep them fresh. And then it also means that when we do want to work with them on more meaningful root-cause analysis work that it's not novel. They're not surprised. So I think that, that has been a real key to our data culture and working with our LEAs, and we'll keep expanding on that as we continue to hone in our data priorities.

00:21:06.91 >> So that data that you mentioned, it sounds like a really robust collection that you, yeah, could do a lot more with even as districts get further along and want to do more of that root-cause analysis and go further, so ...

00:21:21.93 >> Sure.

00:21:22.57 >> You've really set yourself ...

00:21:23.86 >> I would say just as a kind of step feature, the risk analysis tool that we use is really designed initially to help make decisions about the type of monitoring that was going to be conducted with an LEA. Really, how at-risk is their system of special education?

00:21:41.93 >> Mm-hmm.

00:21:42.27 >> Are they going to be able to implement special education in the way in which the IDEA intends so that students receive the [Indistinct]? But we have built on that, and so that was kind of a step to get us to a broader data culture.

00:21:57.52 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at IDEAdata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.66 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.48 >> Hey. It's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.64 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.67 >> Thank you for listening to the second part of our episode with the Maine Department of Education. The team from Maine will continue sharing how they are working to improve the quality of their IDEA data and also use their data to make improvements.

00:00:41.58 >> So we would be remiss to not include our Federal Monitoring Team. Colette?

00:00:46.68 >> Yeah, and you ask about a data movie, and I have to say, I'm kind of stuck because I'm in a bit of a panic because, as always, I have to go after Lee, which I hate because Lee tells a great story, and I always get stuck going after her. But that's okay. I'll do my best. Hi. I'm Colette Sullivan. I am the Federal Programs Coordinator. I am responsible for the team that does the supervision and monitoring across the state as well as professional development, technical assistance and a variety of other topics that need to be completed. I joined the department initially as a member of the monitoring team and then moved into this position about 3 years in. And because initially I was not in a position where I really needed to give the larger data story a lot of thought because my supervisor was doing that, when I moved into this position, I spent the better part of my time in this new role just trying to learn exactly what the data was, what was available. How was it used in that larger sense, how it was impacted and how it impacted the other teams? And Erin really worked to break down silos so that we were all talking to each other about our data and how they all work together, and it was really great. And I remember very clearly the first time Sean asked me for data for the SBP/APR, and I started writing my resignation letter right around that point. But I stayed because he was wonderful.

00:02:22.11 But once COVID slowed down, one of my immediate priorities was really to start looking at the SAUs in a way that was wider, I think, than just across their own individual cohorts or their own individual monitoring sessions, if you will, because I really wanted to help them understand that it was important to see this process in a much broader way, in a bigger way than just a federal requirement that they had to do to just get through it and close their cap. I really wanted them to understand this as a period opportunity, which was something that they really had worked on prior to me, but I really wanted them to see that this needed to be a constant work for them, not just every four years, we're coming in. So one of the things that we started doing was including their previous summary. We call it their summary of finding, from their previous cohort session, their previous monitoring session and really having conversations with the directors about their previous findings as compared to their current cycle, if you will, because we wanted them to see what ... How did their data change? What went up? What went down? And it wasn't my intention to do any sort of root cause analysis discussion. But it sort of lent itself to some very rough beginnings around that, and that information, of course, was very SAU-specific. But we found it was really helpful, and we want to continue doing that. We are ...

00:03:54.73 For Sean, we are trying to pick his brain as much as possible just to really understand. I really want to understand the SBP/APR more. So I'm trying to have lots of conversations with the data manager because the more involved I get in this position, I really become more excited about that process and really what my role is in it, and how can I best present the data that I'm responsible for, my B11 ... not my, but the B11 and BB13 data in a way that really represents what we're doing and how we're working with the field to represent that. So that's sort of broadly how we're using some of the data that we have.

00:04:39.21 >> Lots ...

00:04:39.87 >> It's a few this you're doing.

00:04:41.50 >> Lots of data being collected. It's fascinating to hear and just how you're using it, too. It's not just collecting it for collecting its sake but really using it to improve things and ideas for topics to focus on.

00:04:55.33 >> Yeah, I hope so.

00:04:56.18 >> That's really neat.

00:04:58.71 >> And Mary is our special ... She's not special projects. She's state agency clients and programs, and I'll have her describe what she does for the department.

00:05:09.71 >> Everyone, I'm Mary Adley, the Coordinator for State Agency Programs for the Maine Department of Education's Office of Special Services and Inclusive Education. My movie reference is "Blade Runner." We're using data to make sure students with variabilities don't become missing, as a main character in this movie does. The state agency program team supports Maine's most vulnerable students, who are in the care or custody of another partner agency, DHHS or Department of Corrections, and includes, for example, children who remain in their biological parents' custody but are placed through a DHS referral process into residential treatment because of their medical and clinical needs for additional supports and services in the home and community settings. Regardless of their educational placement by IEP teams on the least-restrictive environment continuum, these children's data, for example, attendance assessment, disproportionality, et cetera, is all captured through their school enrollment with the SAU that's responsible for their provision of fate. We are ... My team is excited about our partnership with the federal monitoring team to evaluate the Fidelity of programming for youth who are in out-of-unit placements that includes ... In Maine, we call them special purpose private schools. There's also educational programming provided for youth at the Maine Department of Corrections Facility and a variety of temporary crisis stabilization placements.

00:06:44.42 A general system of supervision and monitoring data elements for special purpose private schools is outline in our Maine Unified Special Education Regulations, and it includes both quantitative and qualitative components with a similar process used by my team as is used by the SAU monitoring that occurs by the federal monitoring team. And I will close by saying, as my colleagues have already said, collaboration with thought partners is critical to our work of data-informed decisions and instruction. So now, I'm going to turn it over to my colleague, Dr. Roberta Lucas, to introduce herself.

00:07:24.70 >> Thanks, Mary. I'm Dr. Roberta Lucas, and I am the Child Development Services, CDS, State Director, and CDS in Maine is unique in that we are a quasi-state agency that is under the Department of Education and service children both in part C and part B. So we report all of our data through the Department of Education, and I share our data with Sean as he writes the SBP/APR. And I have a data manager, Ariana Whiting, who is also on this call, and she gives me all of that quantitative data, and then I collect a lot of quantitative data throughout our sites. Currently, we have seen an increase in referrals in our children and in our infants and toddlers. In part C, we have h about a 72-percent increase of children with IFSPs, which means that we are approximately seeing a 51-percent increase of referrals year over year. And then in children in part B, we have 62 percent of children on IEPs, which means that we have about a 50-percent increase year over year of children being referred, 3 to 5-year-olds. Maine currently works with a lot of SAUs, and I have been trying to build a repository of SAUs to help service children. Right now, we have about 20% of the SAUs.

00:09:04.89 That's about 45 servicing children in this age bracket, 3 to 5. And they're providing speech and language, OTPT, some BCBA services, and they also help us with transportation to and from public preschool. Because Maine is unique, it also doesn't have universal pre-K. So the data that I get from Ariana helps inform the program needs that we have for children, the capacity needs that we have for staff and how we determine best opportunities for the children that need either an IFSP or an IEP. I just love the two quotes. One of them is from "The Princess Bride" as she enters into the fire swamp: "We'll never survive." And then I answer that with "Finding Nemo," when Dora says, "Just keep swimming," because those are the two things that keep me going, is that I ... Sometimes I feel like I'll never survive, but I just keep swimming.

00:10:14.63 >> Going ...

00:10:14.84 >> And Ariana and my whole team in CDS helps me do that and is behind me. That's me, and I think I'm moving it onto ... I'm not sure.

00:10:27.27 >> Yeah, no worries.

00:10:28.36 >> Okay. Back to Erin.

00:10:30.71 >> Right. So, Amy, you can see why, when I was thinking about who to invite on this podcast to tell the Maine story, it's all of us, right?

00:10:38.86 >> Mm-hmm.

00:10:39.56 >> We are all interconnected, and we cannot ... We were all the puzzle pieces of OSEP. My movie is "The Goonies," because that's what this team is like. They're so ... We're so ... We have such a fun time talking. These are serious topics, right?

00:10:54.15 >> Mm-hmm.

00:10:54.37 >> Mary Adley, serious topics that we have to talk about, and we just ... We are passionate about data, and we want to ... As Sean said, knowledge is good. It's great to have all this data, but if you can't use it to change what's happening in your landscape, then it's not relevant. So that's really our next focus as to how these data will support each other. Our big mission right now in Maine, as I said, is inclusion, and we're all looking for that to change that culture in Maine of belonging and access for ... We think of ourselves as social justice warriors, and we use data to tell the story of how this isn't going well and how we need to increase inclusion. So we were very excited to be asked to talk about this. We talk about data a lot, as you can imagine.

00:11:51.60 >> Yes, we certainly have a lot of data and interesting stories to tell from just the whole team in all different ways. So I've really enjoyed listening to how you all work together and improve the quality of your data that then you can really take it and use it and make improvements. So thank you.

00:12:15.13 >> Yeah. Yeah, we really appreciate talking about this. Like I said, I'm just so grateful for this team and our shared passion and commitment to this work.

00:12:24.21 >> Mm-hmm.

00:12:24.43 >> It's very ... It's a very exciting place to be.

00:12:28.86 >> Yes. Well, passion definitely shines through. So really, really appreciate it.

00:12:35.67 >> Yeah. Are there any other questions you have of Maine?

00:12:39.33 >> No. I'm still trying to absorb ...

00:12:41.15 >> I know. There's so much.

00:12:42.32 >> ... everything. There's just so much, yeah. But really enjoyed hearing about it and just the variety and just all the different things you have going on and how you're working together, is just so exciting to hear about because I think that's ... That can be unusual in states. So kudos to all of you.

00:13:03.99 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the podcast page on the IDC website at IDEAData.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.66 >> You're listening to "A Date With Data" with your host, Amy Bitterman.

00:00:07.48 >> Hey, it's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.64 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.71 >> Well, welcome to "A Date With Data." On this episode, I am joined by a whole group of folks from the Maine Department of Education, and they're going to be talking about how they have been working to improve their IDA data quality and also really use their data to tell their state story, so I'm so excited to have all of you with us. And I'm going to start things off with Erin Frazier, the State Director of Special Services, birth to 22, and she's really going to kind of set the stage and tell us about Maine's IDEA data priorities.

00:00:57.05 >> Well, thank you so much for having us. I have to say that I brought my entire team here because we all have a place in data, and we are very excited about data. We have a lot of XM and enthusiasm about how we are moving forward with our data journey, and just to set the stage a bit, I came to the Department of Education right before the pandemic, and so about 4 months before the pandemic, and so my first 2 years were really focused on that crisis that occurred in education and specifically in special education, and so it's been more recently that we've been looking at data and looking at our SSP/APR data to look and goal-set for our state. We want to use data effectively in Maine to tell the story off special education in our state, especially the story of inclusive education and how we're doing with least restrictive environment, and we are just very, very excited about all of the data and how we're going to be using it moving forward, so with that said, that sets the stage for you. I'm going to pass it to my Data Manager. My team will introduce themselves as they speak.

00:02:09.13 >> Hi, so I'm Shawn Collier. I'm the Data Manager. A lot of my work has been guided by a pretty powerful quote by Emil Faber, who, as you may know, is the founder of Faber College if you've ever seen the 1970s movie "Animal House," the famous documentary on college life.

00:02:27.61 >> Right.

00:02:28.85 >> And so in the beginning of the movie, we see this picturesque view of the campus, stately buildings, and then the camera pans onto a statue of Emil Faber and then focuses on his famous quote that's inscribed below with the profound words that he intended to guide all the students who passed through the halls of Faber College, which is, "Knowledge is good." So with this profound quote ... Okay, so it wasn't a documentary, and I'm making a joke, but it actually does relate to my work and my focus, so for my part, I'm primarily focused on the most fundamental aspects of building a healthy data culture, and I think there are two main areas. One is data quality, and the other is data use. So my background in cognitive psychology research and teaching, because of that, I look at my work through kind of a research and science lens for the most part, so I see data quality and data use as relating to two main research areas that they call basic research. Sometimes it's called pure science. And the other is translational research or applied science. Basic research is understand what is. What exists in the world? How do things work? So this is the Emil Faber "knowledge is good" level of focus.

00:03:45.76 This is where we want to ensure that the data we gather and report are accurate, and so this is where we put a lot of work into data quality, and then we have the data-use level, which is translational research or what they call applied-science level where we look at how the data can be applied to improve outcomes, and translational research data use has to build on the pure research that focuses on data quality. So the data-quality level is my primary focus right now, and this involves a number of things. We're critically reviewing our trainings. We're looking at the needs of folks in the field who are entering and coding the data. We're critically looking at data definitions to make sure they're clear, making sure they're communicated in those trainings, and we're trying to be more proactive looking at the data that's being collected as it comes in. For instance, we're starting to be more proactive about looking at those year-to-year differences and following up with the districts instead of waiting for OSEP to issue those year-to-year reports. There's also a lot of focus right now on using those IDC data protocols to make the data processes explicit. I think this is one of the most single most best things we've done at that fundamental level. They're really great tools. I think we have all the 618 data collections documented using those protocols now.

00:05:09.21 >> Great.

00:05:09.50 >> And we'll be working to add to our APR indicators next. I'd also like to include in those protocols how the various data sets are used. For instance, a lot of child-count files are used for disproportionality, and I think that even if most people involved in those protocols aren't directly related, involved in those analyses, it's a stronger data culture when everyone's aware of their use, and some might even be prompted to think of ways they can contribute to those analyses in their own role. Let's see. The data protocols have been especially helpful because we have a lot of new folks on the data team. We have a brand-new [Indistinct] Coordinator who's climbing a really steep learning curve, and we have a brand-new position also, Data Quality Coordinator, and that person's role has been great at revising the trainings and has been laser-focused on data quality. So, yeah, with the help of a lot of talented folks, we've been building a data culture with a strong focus on data accuracy and reliability.

00:06:15.74 >> Wow. Well, it sounds like there's a lot of really great working going on, and I liked how you talked about the kind of the quality piece, but then it's really a building block to data use. You really have to have the quality there in order to have usable, meaningful data.

00:06:31.32 >> Oh, absolutely.

00:06:32.89 >> Thanks, Shawn.

00:06:34.25 >> Next, we have Tracy Whitlock. Go ahead and introduce yourself, Tracy.

00:06:38.47 >> Hi, Amy. Hi, everyone.

00:06:39.83 >> Hi.

00:06:39.89 >> I'm Tracy Whitlock. I coordinate special projects out of the Office of Special Services and Inclusive Education. The work I do includes our math project, which is part of our ESEP. We do PBIS work, transition work, dyslexia work, but our biggest work and projects that we're doing right now relate ... All of those relate to data, but in particular, this one is focused on data. I can't help but think of Shawn and, "Knowledge is good." For me, I think college movies ... I can't help but think about "Monster University" instead and just keep saying that we're okay, but okay with data and thinking about, what is that we know about the data that we have, and what do we do with it? So we can talk about the data. We can look at the numbers. I think in state government sometimes and in other organizations we might admire the data, admire the issue that you find, but what do you do with it? And so one of the things that we're really trying to move towards is thinking about in particular our LRE data, so we're looking at how our data has remained flat over the past 10 years, and so if we know that from the data, what are we going to do about it? Do we just say, "Well, that's the number, and we're not going to do anything about it?

00:07:56.54 We're just ... have conversations about it, or what programming are we going to put together?" So under Erin's direction, we are working on developing inclusion programming, so with our LRE data thinking about, what is going on in the state, and how do we support the SAUs' ... In Maine, they're called SAUs ... districts to develop their LRE practices so that when they are working with individuals, with students and faculty and staff, how do we get them to be thinking about inclusion? So that is one of the big areas that we're working on related to that. It's also thinking about data. Think about data in terms of the data that we get from our process of, what are our indicators? But there's another level of data that we get as well when Erin has weekly meetings with every special-education director in the state, and we hear what they're having to say, that we are having conversations with our partners out in higher education and in other organizations working with our Maine Parent Federation to find out what's happening with families. How are they feeling about what's going on in the state? So I think for me ... I know Shawn has really fancy words for it, but I kind of call it hard data and soft data, so we have the hard data that we get from Shawn and from the data team, but then there's that soft data. What are we feeling? How are things going in the state?

00:09:22.99 Going back to the idea of inclusion and really working on our LRE data and relating that also to our transition data and thinking about, what does transition need to look like? In Maine, we just in the past year and a half hired a transition specialist, and it has just changed the way we're thinking and having conversations with the field about transition, which ties back again to our inclusion work of, if we're not including kids when they're in preschool, we're not including them in elementary school, middle school or high school, what happens to those individuals, those students when they go into employment? They don't have those opportunities because they've never engaged, or they've had limited engagement with typically developing peers, so Erin will tell you that these are civil-rights issues, and we are passionate about changing our LRE numbers, and again, we have the hard data and that soft data to really have conversations about, how do we have kids included in every single aspect of school? And that's really what our focus is right now and so that they can be included when they transition into adulthood and postsecondary opportunities and employment opportunities and the like.

00:10:32.35 >> Thanks, Tracy. I can't wait to hear ... Are each of you going to have a movie that you're going to use to associate with your work and data? I hope so. I'm excited to hear the rest.

00:10:42.17 >> Well, let's hear from Leigh, who is very excited about dispute resolution. She gets very excited about the data and dispute resolution.

00:10:51.39 >> So hello, everybody. I am Leigh Lardieri. I am the Dispute Resolution Coordinator on the team, and much like what Erin said, the question that intrigued me the most was the one that was asked about, what are you getting excited about? And it's everything. And when I work with the team of people that I'm with, I think of "Rudy," the movie with the college reference to Notre Dame and football and the fact that the character was so passionate, and he had a goal in mind, and he met his goal, and it's just a wonderful, positive story. And so what Erin asked us to do within our individual teams was to talk to our colleagues, my two colleagues on my team, and talk about what data priorities we had within the team, so I'm going to briefly share the three we have for dispute resolution, and then I'll break each one down briefly knowing that we have to stay within a time frame. So the first of our data priorities is to use the dispute resolution and technical assistance data that we collect to help inform the technical assistance including ... That would include day-to-day and protracted technical assistance and professional development offerings by our special projects team members and our federal monitoring team members in connection with dispute resolution. We're really reaching across to Tracy's team and to Colette's team and sharing this information.

00:12:27.96 The next one would be to further explore the relationship between our facilitated IEP program and dispute-resolution filings at the LEA level, so this facilitated IEP program in Maine is still in its infancy. It's a little bit over a year old, and so we're really still looking at whether or not it might be too early to tell, but we definitely can extract some things from it that I'll describe in a few minutes. And so then our third data priority for the future is to use dispute-resolution data related to filed requests to inform regulatory policy changes, so that's a really ... a bigger issue. And without ... I would ... Again, I'll say, without Erin's support, without the support of the team, I don't think we could've gotten to this place as quickly as we have. So I'll talk a little bit about the dispute resolution, the TA data we're collecting, so not only do we utilize what is typical of what due process has to provide to OSEP in terms of the data that we collect every year, but on top of that, we have developed a system where when every single one of our colleagues on our team is on phone duty and collecting information from the field, we are putting that into a database, and it updates in real time, and as we're talking today, 512 communications, either phone calls, emails, have been documented in that database.

00:14:07.28 >> Hmm.

00:14:07.47 >> And the trends that we're getting from callers and stakeholders in the field, emails we're getting from them report concerns about their children with autism getting services or ADHD, their children who have behavioral issues and parent-school communication and staffing shortages. Now these folks are just calling for technical assistance. They don't necessarily turn into requests for filings for due process. Some of them do, but the way that I look at it as a big picture is, when folks make requests to our office and they file requests in Maine, it's roughly about 100 requests a year, and that's combined, and then our number of FIEP requests added to that. So right now, we're almost ... When you combine all of that, CIs, mediations, hearing requests and then your FIEPs, we're about at close to 140 cases right now, but when I look at the cases, those individual cases kind of tell you the weather in those communities or those districts, but if we want to look at the climate around the state, then we need to really branch out and look at those TA calls, and that's giving us kind of a pulse on, what are people thinking? Why are they picking up the phone? Why are they sending us these emails? And so now we're really gathering that information, and we're able to combine that with what we already have, and it's just creating added value to the data we already have and informing what our next practices are going to be.

00:15:56.53 So when we move on to the facilitated IEP program specifically, questions are ... that come up for me are, which LEAs are using facilitators? Are the results favorable as reported by the participants? Because we do ask participants to fill out evaluation forms. We never ask them for any specific information about what's discussed at those meetings, but just in general, how did this process work for you? And then really to look at, is the use of the FIEP program in specific LEAs correlated to a reduction of requests for complaint investigations, mediations or due-process hearings? And so we're going to be looking at connecting those dots to see if it's having that impact. And as I mentioned before, it might just be too early to tell, but over time, we can certainly start to look for those trends, and so what we're essentially doing with the phone calls and those emails is we're really taking that qualitative information that is coming through having a phone call with a parent, giving them space and time to talk because, as you only imagine, as you can only imagine and you already know, they're calling because things aren't going well for their child and their family, and so you're taking that qualitative data, and you're quantifying it in a usable form, which is updating in real time in that everybody on this screen has total access to that because we keep all of this data in one place in terms of access for all of us as team members, so we can pull it up quickly and talk about it.

00:17:46.53 And then for the future, just looking at and monitoring bills, people on my team are much better at doing that stuff in the legislative stuff than I am, and so I look to their expertise, but that's just another point, an important point for us to be able to use the dispute-resolution data to inform regulatory and policy changes.

00:18:10.75 >> Wow, you seem to be doing a whole lot with that data and collecting a lot of it, so I'm excited to hear what you learn from some of those analyses that you have coming up.

00:18:22.46 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.93 >> Thank you all for listening to part two of our special live episode of "A Date with Data" with OSEP at IDC's ii23. Please check out episode 28 if you missed part one, and thank you for listening.

00:00:39.01 >> Sounds like data use, I'm hearing a lot of that, which is definitely something I've heard from talking to states and through the podcast, and I think it's just exciting that we're sort of ... been able to ... There's still obviously data quality issues that states and OSEP is grappling with, but it's just exciting that we're starting to now see kind of the light at the end of the tunnel almost and feeling like we're in a place where we really can start to use and push our data out. And that's definitely a trend that I've been seeing, and I do want to ask you all about kind of the trends that you're seeing in terms of the SPP/APR, kind of digging in a little bit with some of those indicators. There was a lot going on with clarification this year with a number of indicators, and I think Christine will maybe shed some light and talk a little bit about that process.

00:01:29.07 >> Sure. So I'll start with 8 and 14, and I'll have to look at my notes because we see lots of things, and I don't want to fail to mention something when we have the opportunity to sit up here in these comfortable white chairs. But I would say for Indicators 8 and 14, generally we are seeing states ... And you're saddling in to the information collection, aren't you? This is the second year. We had some enhanced requirements, and you're saddling in to what you have to give us, but I think generally folks are doing a better job of describing the strategies that they're going to use to increase response rates, especially for those underrepresented groups. The underrepresented groups part, that specifically underrepresented groups is a place where, first go-round, everyone may have not attended to that in the same way, may generally talk about strategies but not focusing in on those underrepresented groups, and obviously that's very connected to the executive order on equity. So we are paying attention to that, but states are generally doing a better job there. Through our work with OCDO, which is the Office of the Chief Data Officer, we recognize that nonresponse bias is a complex concept, and you would be surprised at how much work we've been trying to do internally just to get better at understanding what it is and also leaning on our TA providers. So I think this is a place where we know that it's going to need some work.

00:02:51.86 It's going to mean that we're going to need to connect with our TA centers, and just moving forward, we also have to think about how we're going to strengthen data in this area to make sure that we're asking the right questions. Because when we go back now and we look at, let's say, the instructions in the measurement table, or all the things that were put there a few years ago, is this going to get at what we're really trying to get at and what the intent is? And it's through submissions we're seeing, well, did we miss the mark a little bit? Should we be asking different questions? How are we going to get at the intent which is making sure that, ultimately, this is about making sure that your families for 8 ... and then obviously 14 is looking at what kids are doing after school, but especially for 8, the families you don't hear from, how do you know what their stories are? How do you know how they feel? How do you know what they think? And just really getting at those voices that may be silent and not present and bringing them to the fore, and that's ultimately the essence of it but through some pretty complex processes. But we recognize nonresponse bias as a place we have to do some more work, have some greater understanding and continue to connect. I think with Indicator 17, what we're seeing is states, over the years, have made some considerable changes as it relates to infrastructure.

00:04:07.48 I think a place where we see opportunities is around implementing evidence-based practices beyond just sort of this very small group. I don't know. What are we, 7 years in, 8 years into the SSIP now? So thinking about how you can scale some of those practices ... I recognize things have happened in states, hashtag COVID and other things, but just sort of how we start to look at that scaling. The other thing is measuring implementation, so that's another place where we see lots of opportunities to support states in that place, but I think generally folks are trying to pick back up after COVID. There's some states that have changed their SIMR, certainly North Carolina talked about that yesterday, but I don't want y'all to think that we've forgotten the SSIP in all of the other bustle of activity. We are still reading your submissions and thinking about how to best support you because a lot of you are doing tremendous work in that space.

00:05:05.78 All right. I left Indicator 4 for last because I recognize it might have been a pain point during clarification for you guys, but I think I told you during that call that this was a great opportunity as ... that the administration has a focus on discipline and exclusionary practices and put out guidance last year. The guidance referenced our expectation that states have a reasonable methodology, and that now this year we flagged we're going to have to start looking at the reasonableness of methodologies. And through initial submissions we recognized that there were places where we didn't have enough information to be able to understand the system fully, so we asked for some more information. There were places there ... And I flagged this during the national TA call, but I'll say it again here. I know that for many of you, especially as you ... some of you are new. Many of you are new. It might be a cut-and-paste process year-to-year, and you've provided language to us that was generally the same, and here OSEP came, in the year of the universe 2023, with this laser focus on this indicator and is now saying, "Mmm, we're not sure about this.

00:06:18.56 I know you told us this for the past 9 years, but we're not really sure that this is consistent, et cetera." And I know that that probably caused a lot of discomfort, but I would say that I was truly moved by the conversations that happened during clarification and the responses we got post-clarification because generally, across the board, states were able to explain. Many cases y'all were doing it, but it just wasn't coming out in the words on the page that they were able to provide the information we asked for and give us a much better picture of what was happening and clarify things where we, at first, were like, "Well, we're not even sure if this is right," so pat yourselves on the back for that. Might have been a pain point, but it is an opportunity, and you guys did a lot to be able to clarify some things for us. Moving forward, I know all of you want to know what we're going to be doing with all of this, and I'm not going to answer that question fully here, but I will say that we strive to continue to understand your methodology fully and have a good picture of what's happening in a state. I'll stop there for now. Thanks.

00:07:30.81 >> Okay, just have to make sure I'm asking the right questions, and maybe we'll get a little more out of Christine. I'll work on it.

00:07:39.06 >> No pressure, Amy. It's got to be the right question.

00:07:41.26 >> I know. I know.

00:07:41.54 >> Oh, wait, I will say this. I will say this.

00:07:43.11 >> Okay, cool.

00:07:43.46 >> I will say this.

00:07:43.96 >> That already worked.

00:07:44.42 >> All y'all know me already. You know I'll ... I'm like a ... shiny things. I will say two things. One is, yes, we recognize that these data for Indicator 4 are lag data, and they were therefore impacted by COVID. So I do recognize that when we have questions, when no LEAs were meeting your end size, all these other things. I don't want you to bash me over the head, "Well, Christine, it was COVID. Kids were at home. We didn't suspend our" ... We got it. I know that. I know that. I know that. You know that. We know that, but we still have to flag for you things we might be looking at moving forward, so just take that in the spirit of that. The other thing I will say is the whole issue about a threshold and the median. I know a lot of you data managers are going to try to hold me hostage about that at some point in time. Mary Corey is looking straight in my face. I'm going to say this. I'm going to say this. Any time we make a decision ... And I've had this conversation with many of you in different places. If OSEP has to make a decision about reasonableness of anything, either we have to have a bright-line standard where we are going to say, "This is it.

00:08:58.59 You can't cross this line," or we have to look at what all states are doing and then figure out, "Okay, who are outliers? And let's deal with that." If you have a third way that we have not thought of, come and tell me, but ultimately that's ... Those are sort of the buckets we have to deal with when we think about reasonableness. So I know that whole thing, media, whatever y'all are about to jack me up for, but nonetheless, know that we have to operate in two spaces. Either it's a bright-line standard, or we're looking at everybody and figuring out who the outliers are, so just know that. So take what we say within that spirit, but I'm pleased with the work that has happened not only internally, in terms of our analysis, but how the kinds of conversations I heard through clarification. There's some of you that brought back up PowerPoints from 2010. Luckily, we were still saying the same thing. There are those of you that have IDC's Resource dog-eared at your desk. I have folks on my staff who do, and honestly their resources can answer just about any of the questions, so I'm pleased that we're being consistent over a decade in time, and I'm also grateful for IDC for all of the work that they have done in this space because they can continue to support. So don't be afraid of Indicator 4 and what's going to happen. It's really ...

00:10:15.57 It's an opportunity, and ultimately it's about what's happening with kids in that space, and that's what we're all here for so ...

00:10:21.97 >> Yeah.

00:10:22.80 >> You're not going to get me to say anything else. That's it.

00:10:24.15 >> Okay. Okay. I will move off from ... The spotlight is no longer on Christine for a few minutes. Richelle, what about 618 data? Any trends, things of note that have been happening over the last few years that you want to ...

00:10:38.02 >> Yeah, we can come up with a few things. I think the biggest thing is, is that we're seeing kind of the recovery out of COVID, right? So the data ... We're not seeing data quality issues that are likely from COVID. We're seeing, as you actually look at what the data are, they're starting to more reflect 2019 than they did 2020 or 2021, and so you can kind of see there was a blip, and now we're trying to come out of the blip. I think the big thing ... and Kristen kind of ... Christine? Kristen? Christine.

00:11:15.33 >> I can be Kristen.

00:11:15.95 >> We ... I can be anything that ends with an Elle, usually. But I think what we ... one of the things that she said ... she didn't ... You didn't actually say the words, but I know you meant it, incremental improvement, and I think we have seen that in all states for years and seeing how things incrementally improve. And I think knowing how much turnover there is in states, I think it's a really a testament to you and to our TA providers for the support they're able to do with data processes and just the general TA that's provided, to see what a consistent level of data quality there is. That's not to say that everything is perfect, and I know zeroes can be such a problem, or the lack thereof, and so everybody counts or nobody counts, but it's really interesting looking back over the years to see where we are. I won't say that the more people and the more technology that we get, the more data quality errors that we're going to find. That's not necessarily a bad thing. It's kind of a good thing, right? Your data is going to get cleaner because you have the ability to go out there and find more errors. You may not like it from a data quality ... looking at your data quality rubric standpoint, but I think it's definitely good for the data quality.

00:12:45.56 So that's really it, and again, as I've said many, many times before over the years, you guys do a remarkable job of getting us really high quality data at the first time, so that's why I'm not as nervous about EDFacts modernization as some people are because I can go back and I can look at the history of the data quality that's been submit ... of the data that's been submitted and look at the quality of that and know that, yes, there are errors here and there. Yes, there are blips, but for the most part folks in this room, all of the states submit really high quality data on that very first submission. So I have faith in you all, so please don't prove me wrong.

00:13:30.45 >> All right. So now we're going to shift over again to Gregg, and we want to hear about DMS. And Differentiated Monitoring and Support is a big deal for all states, and I know with IDC, we've been doing a lot of work with states to make sure that their processes are documented so that they're in a good position and prepared for DMS and also helping states complete those protocols that are available for each of the framework components. So I'm wondering what other advice do you have for states that have upcoming visits or still are a few years out even, what they can start doing now to get ready?

00:14:09.22 >> Okay, thanks for that. Yeah, so DMS is one of our three big buckets along with SPP/APR and grant awards, and you may have noticed that attached to the conference application is a page with a number of linked items, and I checked, and it is there, each of which should help you with various aspects of DMS. So if you haven't, please check that out. A few tips and suggestions: We've been featuring DMS on our monthly TA calls for a couple of years now. Those calls have all been recorded and archived. So one of the links, if you click on there, will take you to all of the monthly TA calls. Not all of them are on DMS, but you'll be able to find the ones that are to go back and review that. I would urge you to review and begin populating the protocols that our teams use when they engage with a state. Those are all also linked, and it's a good way, even if your state is not part of Cohort 2 which was the upcoming cohort. Even if you're Cohort 4 or 5, it's not a bad thing to begin populating those protocols as a way to assess your own systems. Use the document review and request template to begin organizing and preparing your documents, and I think that's really critical for states that are going to be monitored in this coming year. It's a major task.

00:16:03.68 We recognize that it's a lot of effort on your part to provide all the documents that OSEP is requesting and organize those, so use the template to begin doing that. And please don't ignore our OSEP-funded TA centers who are a tremendous resource for all of you. Set up a time to meet with your designated TA provider, could be NCSI, IDC, Intact, Cipher or others, but they are all ready and prepared to help you with the process of getting ready for an OSEP DMS monitoring as well as helping you through the process. Lastly, we have provided a link to a presentation that was made during last year's OSEP Leadership Conference, so please take a look at that. I think there's both a video and a PowerPoint there that you might find helpful.

00:17:06.27 >> Great, thanks, and I'll just reiterate what Gregg mentioned. And in the conference app you'll find some resources, some of the ones that Gregg mentioned, so please definitely check those out. And one thing I think I started to see for sure as a result of DMS upcoming is just all the work that states are doing to start just making sure, is all the information available on our website? Even putting together new resources and technical assistance and just everything you're starting to do which is a great, I think, kind of consequence of having this, is all that great work you're doing to try to make sure you're set up for success. So I commend you all on that. I also did want to mention we did collect a few questions from you all yesterday, so thank you for sharing those, and we have ... We went through them and have been trying to make sure a lot of them have been incorporated already in the responses and questions we've asked, but one question, another hot topic back to Gregg has to do with determinations which is ... That is something else that is coming up soon and I know is forefront on everyone's minds. So what can you tell us, Gregg, about determinations?

00:18:14.25 >> We ... Probably not as much as people want to hear at the moment, but let me go ahead and get started. We did a lot of work in the past year to collect input from a variety of stakeholder groups to provide feedback about the current determinations to the extent that they are the correct way to analyze a state's performance across both compliance and results. We got a lot of terrific input. Also, there were a number of written submissions that we analyzed as well. We do listen to you because we think that there are some great ideas out there that we definitely want to avail ourselves of. So taking all that, we have made a few changes for this year, and we're looking at some ... probably more significant changes for FY 2024. Now, as a commercial, an advertisement, tomorrow at 4 o'clock, Christine will be leading a TA call on the 2023 determinations where she'll be able to share a little bit more information with you. If any of you are traveling at that time and not able to access the live call, that, like our other calls, will be recorded and archived, so you can look at it at a later time. Usually that takes a couple days, but by sometime next week you should have access to that call. Anything you want to say about that? No? Going to save it all for tomorrow, huh? All right.

00:19:55.90 >> I think you covered it well, yes. Little things right now, still looking at what happens beyond 2023, and we referenced that in last year's letters when we were saying, "We really want to focus on issues of equity," and gave some examples and looking at existing indicators or anything else we could do new. We flagged it was ... We were thinking about 2023 and beyond, so I think anything more substantive would be coming later, but there probably are a few of interest to you. Listen on Thursday, see what I say.

00:20:30.07 >> All right.

00:20:31.31 >> So stay tuned. I'm sure you'll have a large audience for that one, and one of the reasons really that we started this podcast originally is because we hear from states all the time that they really want to hear other states' stories and they want to know what resources you're using and what are your successes. What's working? So I did want to ask you all if you have any stories that you might want to share, successes where you've really seen states have improved the quality of their data? Christine, do you want to kick that off?

00:21:03.93 >> Sure. I think one that I use often because I was present for the bulk of it, and I remember it well, involves Mary Watson, who is sitting at the back of the room. Mary, this is an example from a state that you have worked with. A few years ago, I had a state ... I was their state lead at the time ... that they were losing points on the 618 portion of the data rubric, all zeroes in different places for different collections, and the state director at the time was not happy about how they were losing all of these points and brought together the data folks within the SEA and everyone in her office, et cetera, and worked with IDC around that data processes tool kit to really be able to map out just sort of the work that was happening and what needed to occur, and it was very evident there, as I think still tends to be the case. EDFacts coordinator didn't really have a good sense of how some of ... How does assessment data play out if I don't submit it on time, or if I don't respond to something? How does this actually affect a data rubric which is a score on the compliance matrix which is a part of determinations? And I think through that process ... and they were able to then ...

00:22:17.33 They were not losing points as much as years went on, once everyone was aware of how these submissions affected a rubric which affects a score on the compliance matrix which affects determinations, not necessarily in a big way, but there is an impact, of course. And through the work with IDC, I saw them change their data quality dramatically as a result of that work. And I believe so much in those data processes, that data processes tool kit. IDC has helped us internally think about how we document our processes and what we do so just to reinforce how incredible they are in that space, but I think Richelle is not worried about ED modernization, but it keep me up at night a little bit because I would say that this year ... So this isn't about to be a success, but we're going to make it a success, Amy, stay with me. I saw more states have issues with 618 submissions, specifically assessment data, and then it was sort of not recognizing until after the resubmission period, "Oh, shucks, our stuff was wrong." And that concerns me because ED modernization will not have a resubmission period, so all your stuff has to be right when you put it in. The assessment data is going to be coming in, in January next year.

00:23:38.27 It's a short turnaround time before it's in the APR submission, February 1, et cetera, but I'm not sure if everyone understands that we use the data as of that freeze date in determinations and to make decisions because those are the data that go through data quality reviews. So when a state recognizes a month after that period, "Oh, my gosh, we completely messed up these data," we are already using that in determinations, and that's ... That keeps me up at night, and while generally across the board you guys are doing a better job, I only need one state to make that mistake and I'm sweating. I'm like, "My world is about to upend," and, "What is this going to mean for determinations if we're using data as of the freeze date, and now the state is saying it is wrong?" So I really need for you guys to be very ... be thinking about what we can do to support you in ED modernization. So if there's something you need from us to make sure that you're able to give us data, and as of the date you give it to us, it is spectacular, and you can stand behind it.

00:24:40.26 Please tell me what that is because I feel like all y'all are my children, and I worry, and then if the data is wrong and we're using it, and now somebody needs intervention and because of ... I can't have that, so you tell me what y'all need to make sure that you're going to be right on target. Can y'all do that for me? Okay. That's what I need from you. Does it keep up Richelle? It keeps me up at night.

00:25:03.66 >> I sleep pretty well. Anyway, I just kind of wanted to reiterate off of that, we are working on trying to come up with some solutions and some ... Working with IDC to come up with some ideas on what we can do to support you so that you can really rest assured that the data ... that you're seeing things the way you mean to see them. And so as Christine just said, if you have ideas on what a resources would be, what it is that you need for us to be able to help you ensure that the data, particularly the data, the assessment data that are going into your SPPs and APRs are top-notch, ready-to-go, best-thing-you've every done before in your life. Please let us know because we have some ideas, but there are ideas, and it may not actually work for you. So if you have an idea, I think that would be super helpful. So thanks.

00:26:01.27 >> I was going to put a plug in for IDC's EDFacts data manager peer group, if you're not already part of it or don't ...

00:26:08.12 >> Man, that's on my notes for later.

00:26:09.68 >> Oh, I'm sorry.

00:26:10.30 >> That's okay.

00:26:10.89 >> I'm sorry, Richelle. I stole your thunder, but we'll say it twice. It's that important.

00:26:15.08 >> Okay.

00:26:15.81 >> So definitely check that out as well, and also to that end too, I think ... And IDC is always listening through our different peer groups, of course, and just the work we're doing for those success stories, and we'll definitely want to highlight in particular with EDFacts modernization if there are strategies that you're using in your approach, for those states that are getting it in and aren't thinking a month later, "Oh, no, there was a problem," and how you're really making that happen. We'll definitely want to highlight those and bring those to everyone's attention, so you can learn from each other. And any other success stories we want to ...

00:26:52.06 >> Well, I was going to talk about a data processes one, but you stole that, so I don't know. I'm at a loss, but, no, I ... just to go on, the data processes ... I don't know. Is everybody in here aware of the data processes tool kit and the working with IDC on that, any? Okay, yeah, people seem to be, so that's great, and if you haven't, I think that it's really important that you do it. I know we've had, like I said earlier, a lot of turnover in different states, and so my success story is not necessarily one story, but it's the stories where folks have moved on to greater things, and where someone has been able to step in and step up, and the data quality has maintained and/or improved, and so that is my one. So I'm just going to say it, and I just did so ...

00:27:42.18 >> Oh, sorry, sorry that I stole something that you wanted, Richelle.

00:27:45.06 >> That's okay. I'm okay with that.

00:27:46.48 >> But I will also say that any work I've seen in states that would count as a success story and them getting to a much better place with data quality has always involved their TA centers. So if you are a state who is not accessing your OSEP-funded TA center, you are not setting yourself up for success, guys. So you need ... They need to be your BFF, your right hand. Don't have a call without them with OSEP. Have them there with you lock-and-step through every process because every story I can think of where I'm like, "Oh, yeah, they did a much better job. They did" ... always involves support from a TA provider so don't mess around and not have them involved.

00:28:30.36 >> What she said.

00:28:33.19 >> Well, thank you for that wonderful plug and, yes, there are many of us scattered about, and all of you know all of us so definitely just ... If there's any time you see us, just grab us, one of us, and just say, "I really need help with this." We are all happy to pitch in at any time with anything you need IDEA data-related. So along with those successes, of course, there are challenges, and we know there's challenges coming up with different changes that are in the works. Richelle, do you want to talk about some areas of challenges we see happening and ways that we can try to support states maybe with those?

00:29:14.14 >> Yeah, so again kind of hitting that same old record here. EDFacts modernization, I think, again, it's the big change. It's the big ... the big piece that's out there, and so I ... As I was working on my notes for this, I was like, "Wow, in order to be good at the new EDFacts modernization and getting everything into EDPass, it's really going to be all about that improved data culture," that you guys have spent the past couple days learning about because it's going to be about making sure that you have those strong relationships with your EDFacts/EDPass coordinator so that you ... everybody is on the same page and that nobody is hitting any buttons early. So that's really ... I want to make sure that that's the big one. We know that change is hard. You know that change is hard, and it's going to be hard for us at OSEP, and it's going to be hard for you in the states, but I have faith. The other big one is that we're all going to need a lot of patience. OSEP is going to need patience with the states. States are going to need patience with OSEP and with ED and with EDFacts, and so we're ... As I noted earlier, we're experiencing delays in getting EDPass open.

00:30:26.35 We don't anticipate that that will impact other due dates in the future, so continue as you've already planned as far as the other collections, but, yeah, we do know that right now child count is delayed, and I don't have a date for you. So if anybody has a follow-up question, I don't have it right now, but I will say to make sure that you're watching your emails for updates. The other thing that I wanted to point out because I don't know whether you realize this, but the system isn't going to run year-to-year checks this year, and so I would still really advocate for you to run the ones that you've built into your systems over the years for yourselves because, number one, we're going to do it again next year, so you might as well just be ready. But number two, I do think that those year-to-year checks, when we first invented them a long time ago, was to actually find the little fat-finger errors, where you hit the eight instead of the nine, or you hit the whatever instead of the whatever but ... So we would really suggest that you keep doing that. One area, another area that we're looking at for support ... I don't know whether you guys are aware, but we posted a notice of proposed priority a couple of year ... a couple of years ago?

00:31:44.25 A couple of months ago looking at a ... basically a significant disproportionality data center, and so within that proposed priority, there are several different questions that we had posed to the field, to you guys, to ask for your ideas around how we at OSEP can support you as far as a data center would go. And so if you can take a moment and work together ... I don't know whether SEDMAG is going to do a big response, but to go back in and look at that and then tell us what it is that you need. That way, we can make sure we're designing not what we think you need but what you actually need so ... And again, I was still going to plug the IDC's Peer Group. So make sure you get involved with those, especially with your EDFacts coordinator. You can just get them around the arm and walk them right up in there.

00:32:36.86 >> We have some more topical conversations with our peer groups coming up today too, so definitely check those out. Christine?

00:32:46.28 >> Just two things in terms of places where I think we have some opportunity. States might need a little support in this area, and we have an opportunity to do that in the discipline space. I would say when informal removals should be counted, as a disciplinary removal seems to be a fair amount of confusion and conversation around that, and we see that as a place where we ... where some support is needed but then also when a series of removals constitutes a pattern and then how that is reported in the 618 data. So I think as it relates to the discipline collection, those are two places where the more we're digging into Indicator 4, where it seems like there's lots of conversation or confusion, et cetera, so I would say that. And then for the SSIP, what's top of mind is the states having a true fidelity measure and a mechanism to collect those data and report those data to us would be a place where that's a ripe opportunity, to think about how we could support states in doing a better job there, but those are the ones that are front and center in my mind at the moment.

00:33:57.58 >> Great.

00:33:57.70 >> And kind of an overarching challenge that I think all of us face is just vacancies and staff turnover from the local level to the state to the federal. I actually just got a text that one of our people is leaving, but it continues. It continues to happen, and we recognize that the data that you get at the state level is only as good as the data that comes up from your LEAs, and it's a challenge. We recognize it's difficult. We also want to commend you for staying on top of that and doing the best that you can despite the challenges that are out there.

00:34:43.07 >> Okay. Oh, I'm getting a 1-minute, yes. So I don't want to end on challenges necessarily, but I think we heard a lot of great information and positives and exciting things to come. I don't know if anyone has any last words before we wrap up.

00:35:02.85 >> Well, I was supposed to answer a question on ... Let's see.

00:35:06.28 >> Oh, LRE? Do we want to ...

00:35:07.42 >> 619 data for LRE, and is there a consideration for no pre-K being compared to universal states? I don't see us doing that. There's nothing that could stop someone out in the world from writing an article, but it's not going ... At this point, for the next 3 years at least, I don't ... It's not a part of the collection. It would be really ... we would have to find a way to have that as a piece of the data collection, and that's not in the cards so ... But again, I just want to thank you guys for everything that you do as far as to support high quality data for kids with disabilities so ...

00:35:40.82 >> Great.

00:35:42.16 >> I ditto what Richelle said, but I would also just add I truly treasure you guys, as our accountability partners, and while I make jokes with you and say your name out on stage, I appreciate the questions you ask, the way that you push us sometimes in particular places if something we're saying or doing is not making sense, and it helps us get better, so continue to hold us accountable.

00:36:10.75 >> And I just want to thank Amy for moderating this chat. This was a new format, and I think it worked really well, and thank IDC for having us here. It's been a pleasure talking with you all.

00:36:21.85 >> Yeah, thank you all so much. We really appreciate this, and you can also listen to this again, if you like, because we will be putting it out on the podcast at some point. So thanks, everyone, have a good rest of the day.

00:36:38.57 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.67 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.50 >> Hey, it's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.65 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.49 >> Welcome to the very first live episode of "A Date with Data" podcast. We are very excited to have OSEP, the OSEP edition today. For those of you maybe who haven't listened to the podcast yet, I would encourage you to check it out and hear from many of you that are here today in fact. And usually on the podcast we have state special education directors and data managers and all of the rest of you that are here today, as well as district and other state staff join us, and this time we're so happy to have a national perspective on IDEA data quality. And they're going to be talking to us about, from that higher-up view, and what OSEP is thinking and OSEP's priorities related to influencing, interacting and improving our IDEA data quality so, so excited to have you all here. So as Julie mentioned, we have Gregg Corr, who is the Director of the Monitoring and State Improvement and Planning Division, or MSIP, also Christine Pilgrim, who is the Associate Division Director with MSIP, and Richelle Davis, who is an Education Program Specialist in the Research to Practice Division with OSEP. And I just wanted to start out by having you each just say a little bit about your role, what you do at OSEP and also just something you like to do for fun when you're not working. And, Gregg?

00:01:53.13 >> Pick on me first?

00:01:53.90 >> Yeah!

00:01:54.71 >> All right. Okay. So my role, as it has been for over a decade now ... Time flies when you're having a great time. So as the title of the division might suggest, we are responsible for monitoring but also for SPP/APR and determinations. I look to my friend here to my left, who is deeply involved in that process, and also grant awards, and we have staff back in Washington who are working very hard to review and approve the grant award applications that came in May 24th so that everybody can get their fund on July 1st. I know you all are looking forward to that. So that occupies a lot of my time. We also do a considerable amount of policy work. We try to squeeze that in where we can and customer service. We do have a customer service team that varies from three to four people, depending on vacancies, but they are kind of our first line of contact with the public. A parent who's upset calls in. They engage with that parent, inform them of their rights, kind of suggest alternatives and resources for them. So those are some of the things we do. Now I forget the second part of the question.

00:03:24.18 >> What do you like to do for fun when you're not working?

00:03:25.92 >> Okay, when I'm not working. Sleep, and over the pandemic, we acquired a puppy, a cute little border collie puppy. The kids had been asking for one for a long time, and I kept saying, "We're never home! We can't do that," but now we're home a lot more. So these dogs are incredibly active. Her ... The dog's parents were working sheepherding farm dogs. We don't ... Do not have a herd of sheep in Capitol Hill, so we ended up taking her out on walks and out to throw the ball as much as we can. So that's a lot of my time and kind of trying to keep the garden looking presentable and taking out the garbage and that sort of thing.

00:04:11.68 >> All those fun activities.

00:04:12.79 >> Yeah, all those fun activities.

00:04:14.84 >> My turn next. I have been with OSEP for 14 years, but in my current role and as one of four Associate Division Directors, I've been doing that for 3 years. I've had various roles at OSEP. I've been a State Lead. I've been ... I called it a party planner, but I was really a Facilitator of Collaboration across the department, and now I'm in this role. As one of four Associate Division Directors, I have what we call a homeroom team. So I lead a group of 10 folks are on ... I think it's about 12 folks are on my team, and we have a certain number of states and all of the Pacific entities, so my friends from American Samoa. Guam can't be here because they're dealing with the effects of a typhoon, but CNMI, FSM, Palau, all of them are on my team as well, and my team is called Team B. I say it's B for best, but this is going to be a podcast living in perpetuity, and then now my colleagues know how competitive I am because I want to be the best at everything. I also provide leadership to the data implementation team, and that is a team that is charged with overseeing our process to review your SPP/APR submissions and issue determinations. I am not a native sort of data person. I've had to learn a lot as I have taken on this role, but I always appreciate how accountable this group holds me and everything I do, especially [Indistinct]. So if I can usually get through them and I'm looking okay and I don't have egg on my face, I know that I've achieved something. In terms of outside of work, I don't know that there's really an outside of work right now because it's our busy period, but I do like to eat really good food. I like to sleep. I like to travel, but I have a broke best friend, who is my 18-year-old child, and she affects my outside-of-work shenanigans all of the time. Right before I came onstage, I got a text. "Can you put some money in my account?" That's all I ever ... "Can you put some money in my account? Can you put some money in my account?" So that's my life. Richelle?

00:06:16.16 >> Hey. So I kind of gave a little bit of it away yesterday, but I like to consider myself Ms. Other Duties as Assigned. So I started at OSEP in the Research to Practice Division with the data team, and I still work with the data team but ... and we collect the data, as I talked about yesterday, so we kind of are that first block. It's where the data comes to, and we're doing that initial data quality and then pawning it off here to Christine. I also work on the Annual Report to Congress. Who's familiar with the Annual Report to Congress? Anybody read it? Hey! You guys are making my day right now. So I've done that now for probably ... I want to say 10 years, maybe 9, not entirely sure. So we're working on the next version. We just published one. So if you're looking for something to do, you can find it out there. I also do some work on the Personal Development Data Collection system which is a lot of fun. We fund grants to scholars, and then we demand that they do service. So they have to teach in the field for a time period, and so we have to collect that information, and so we track that as well, and along the line I've also managed to get myself Special Olympics. Who's familiar with that? Unified Champion Schools, really cool. This has been extremely rewarding for me. I started my career in Richmond City, and last year I got to go to ... They call them Banner Schools, and so one of the elementary schools in Richmond, not one where I taught but still pretty cool, they got a Banner, and so Secretary Cardona came, and he came off of his little bus. It was the bus tour, and the kids were all lined up, and then they played cornhole, that game you play at ... Yeah. Anyway, it was kind of neat watching the kids play cornhole, but it was pretty special to kind of have my career kind of come full circle there, and so I've picked up some other things, work on trying to get the discretionary grant money out the door, project officer to IDC. So whatever needs done, I'm a big fan of making sure that it gets done. In my free time, I don't know whether you gathered this, but I kind of like to golf. Someone mentioned, "She looks like she's dressed to go golfing," and, well, yeah, that's my road trip attire, to come back from golf, after I've played outside, untucked the shirt and put the sandals on. So I'm a pretty avid golfer. I like to get out a couple times a week, so that's my real big non-work activity.

00:08:53.49 >> Great! Well, thank you, and now we got to get down to business. All of you, I know, are dying to hear what is on OSEP's minds. What are you all focusing on? So can you tell us a little bit about some of the priorities now from OSEP's perspective? And, Richelle ...

00:09:13.60 >> Sure!

00:09:13.76 >> ... if you'd like to start.

00:09:15.66 >> I would be happy to start. So right now our really big priority that we're working on is ED Facts modernization. So I know a lot of you in the room have heard of it. I'm guessing that there may be a few that haven't. Basically what ED Facts modernization is doing is moving our review of the data. So the idea is to have high quality data that is usable on the due date. So instead of having that time period where the data is submitted, and then OSEP goes and reviews it and then gives you guys comments, and then you respond to comments and then resubmit the data, the idea is, is that all of that work is going to happen on the front end, and so a lot of processes are changing. Things are being built into a system, and so we're going to use a new system called EDPass. I'll just save it. I don't know what date it's opening. It is in flux. We will hopefully ... Keep checking your emails. We'll have a date for you. Obviously all new things are often delayed, and this is no different, and so that's one of our really big priorities that's going to happen. It's kind of fun looking back at what we did yesterday and seeing how ED Facts has changed things. I expect that I can be back up here in 10 years talking about how ED Facts modernization has changed things, so that's our biggest priority right now. Another big priority is publishing more data. So we have worked on publishing the LEA Child Count data, and so we just published our second round of that. So there are two LEA Child Count files that are out in the public sphere. Next year, we hope to publish an ED ... an ED Facts ... an exiting LEA file. We're also, along the same line of getting data out the door and really trying to influence the public with our data. We're cranking out the Fast Facts. Who's ever seen the Fast Facts, the little one-pagers? Excellent. If you haven't, it looks like there's a hand at every table. So if you don't know what it is, feel free to ask the person that just raised their hand. I'm sure they can hook you up, or you can ask me later. And then perhaps the biggest thing that we do is making sure that data on kids with disabilities is at ... The data is at the table for every meeting that we attend, so that's our big one, so ...

00:11:40.28 >> How about MSIP?

00:11:43.81 >> I think to reinforce some of what Richelle said, I think this focus on transparency. What data are out there? Is it easily digestible? Is it accessible? What are we putting out there and how we're using it more effectively to make decisions? I think with the President's Executive Order on equity, we've looked at a number of our internal processes and thought about how we, as you know, how we could think about ... how we make determinations differently in 2023 and beyond, and we've gotten a ton of feedback from you on that. And then obviously that also plays out in information collections, another thing, and us thinking about policy, et cetera, but I think focus on transparency. Are we getting the right data? How are we using the right data to make decisions? Those are the bulk of our internal sort of conversations and also in thinking about DMS. In Heather and Susan's session yesterday, I sat in although I was worried I would squelch conversation. They talked about how data is a thread through everything, and we also talk about that internally a lot, and while I'm not a native sort of data person, I have learned that throughout my years at OSEP. There can be no conversation about any of our work processes unless somebody is looking at the data and seeing what the impact of the data is. So I think looking forward to DMS, we are doing what we can to make sure that obviously there are protocols for data in the SPP/APR, but that we're really getting some good conversations we are on site with states and engaging with them deeply to ensure that data stays at the forefront and that we are able to break down some of those silos. I think I've been in situations before where people ... Maybe the ED Facts coordinator doesn't really understand what's happening with IDEA data and what happens when that data makes it in the SPP/APR, and we make determinations. So I think some of those conversations on site also help to further that. So I think from the MSIP standpoint, at a very granular level, those are the things that are top of mind for us, but Gregg, as my boss, might have other thoughts.

00:13:33.80 >> Yeah, so thank you, Christine, ditto all that and also would want to add that one of the priorities for MSIP is increasing our internal capacity to effectively use data, and we've been successful recently in bringing on new staff. We actually stole a person from Richelle's data group in the Research to Practice Division. Apologies.

00:14:02.71 >> So he said, "Changing faces, changing places."

00:14:05.12 >> Yes, that's definitely the tune. So we have a data analyst that's a brand-new position, someone with some pretty significant data chops so I think will help us tremendously. We also were able to replace the data implementation team facilitator. That person, who may be here, left about a year ago. Christine has assumed those duties. That's one of ...

00:14:33.56 >> That's more than a year.

00:14:34.53 >> More than a year, okay.

00:14:35.43 >> Yes, sir. Yes.

00:14:36.08 >> You would know.

00:14:37.45 >> Yes, sir.

00:14:38.17 >> Yeah, yeah, but it's one of your many full-time jobs, right? So we've done that, too.

00:14:46.07 >> Sure.

00:14:46.27 >> We are trying to build a data culture in MSIP, and before I left Washington, I texted Glenna Gallo, who is our new Assistant Secretary. Yeah, we're excited about her, too. I just got a text from one of my colleagues saying he was in a meeting, and Glenna was there, and she was so organized and positive, and it was just really delightful to have her there, but she did send me a few things that maybe this would be the best time to share. So she, Glenna, called me up yesterday morning and said, "Do you have a minute?" And she wanted to share some messages that she wanted me to convey to you. One of the things she wanted to share is that she is a self-described data wonk. She loves data, and she said she actually has to hold herself back because she has a real tendency to kind of get way down in the weeds too quickly, go down those rabbit holes. So that's something I thought would resonate with this group. She shared some ideas that she's working on. She's got a Call to Action. One of her priorities is aligning current areas of improvement with values and objectives for the future. So not just what we're doing, but where do we want to go? She talked about some strategies for success. Part of her vision, I think many of us share, is a coordinated seamless system across Part B, Part C, the discretionary programs, Part D and then state and local programs and also working with RSA and postsecondary. She's interested in further building partnerships, engagements and coalitions with our varied group of stakeholders out there. Another priority is learning from the past, data analysis and data-based decision-making, really kind of using data to examine inequities consistent with what Christine just said about the Executive Order for this administration, and then finally identifying and using evidence-based practices that promote access, equity and inclusion. So we'll be hearing more from Glenna, as she will be one of our featured speakers at the OSEP Leadership Conference next month, but this is a little preview of her values.

00:17:15.85 >> Great! So in the spirit of this conference being about data quality influencers and proving your data quality, can you each say a little bit about how you feel like you're a data quality influencer?

00:17:29.05 >> Sure, so I really like the concept of a data quality influencer because it's that idea of making sure that ... I think that you improve data quality by using it, and so the way that you influence data quality is by getting other people to use your data, and so I think that's a lot of fun, and it's great. Hearing about Glenna's comments and her interest in data and being able to ... I'm actually looking forward to bringing data and getting her down the rabbit holes with me but being able to have that data at the table. Obviously we influence data with the collections and things like that and what's going to be collected, but I think the more important place of influencing the data quality is by getting the data out and using it and bringing it into all of those policy conversations. So that's the ... really the biggest place where I see my ability to influence data quality.

00:18:25.23 >> In addition to obviously the formal processes, and Richelle touched one of them, an information collection, obviously we influence what happens based on the data that we're asking states for, but I think when I look at our team and how we work internally, it sort of warms my heart, right? It sort of makes your heart sing how our conversations have changed over time as we are analyzing SPP/APR submissions, and the more we learn, the deeper our questions become internally as we're trying to analyze your submissions, and then we get into a place of, "Okay, well, are we even asking for the right thing? Is this getting at what the intent was? Maybe we need to think about how we change this. How is this going to affect our TA, et cetera?" So I think a lot of those internal discussions then dictate what we want to do when we think about how we're connecting with the OSEP-funded centers T ... That we want to put out and things that we may be figuring, "Okay, well, next information collection or next time this comes up, should we tweak this a little bit?" Because now that we're getting submissions and what we're seeing, is it really ... Are we getting what we thought we would get? Are we asking the right questions? Do we have confidence in what we're getting that it's telling the whole story? So I think that's one way, and that's again sort of our internal continuous improvement process, always looking at the data. Are we getting what we want? Is this what we intended? Is there a better way to do this? I think ... It was hard for me to think of myself as an influencer, but I am going to put it on my evaluation too, in addition to finding Gregg the page. We have been much more intentional about collaborating with our data scientists across the department. We have an Office of the Chief Data Officer, and while they're not steeped in program stuff so they don't really know IDEA and IDEA programs, they're brilliant data scientists, and we've been trying to connect with them more as we are trying to operationalize particular things in our office and make some decisions and sort of pressure testing a lot of our thinking against them, and I believe that makes us stronger as an office when we're saying that we want a particular thing, that we've done this, and that is also an influencing moment. And I think DMS is another place where we influence, right? I know that what we measure, you guys value. I know the types of questions we ask of states, also guide work and conversation, so I think that's another place. I think lastly, one more, y'all may not realize because one or two of you always find a mistake, and then I'm ready to stab myself in the eye at the end of the process, but we do have a very rigorous process for SPP/APR review. There are multiple eyes that see your submission. We take our quality, our QC, processes very seriously. Everyone in the office knows how much I hate a mistake, hate making a mistake. They do happen. Y'all let me know when they happen, and I appreciate that, but it's multiple levels of review. It's across multiple offices. It's us trying to make sure that our office isn't saying something that's not congruent with what OESE is saying about the same data, et cetera. We're very intentional about those things, and I think that's another place where data quality is always top of mind for us, and we want to make sure that we're putting the best foot forward.

00:21:24.19 >> And you can imagine the enormity of that task, when we're talking about 60 state submission Part B and 56, I think it is, for Part C, multiple indicators for each program, and each indicator is broken down into little bits, so lots and lots of data out there. The potential for an error is significant, and I think it's a real testament to the commitment of our team that very rarely do we make mistakes that don't get caught before they go out. So they're doing a terrific job. To add onto what Christine already said, we're trying to make better use of our data across different processes so that, for instance, when a monitoring team is preparing to engage with the state that they're also taking a look at the 616 and 618 data to kind of inform their questions and their analysis and review. We also want to make sure that what we're hearing from states through the SPP/APR is congruent with what we're hearing when we conduct reviews and review documents that the states submit as part of DMS, so making sure that there's consistency across our different processes.

00:22:53.11 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.83 >> Welcome to the second part of "A Date with Data's" conversation with the Ohio Special Education Data team. We are going to continue our conversation about how the team works and collaborates together and also hear about some of the exciting initiatives they're currently undertaking as well as plans for the future. What are some of the initiatives that this group is working on now and that you're really excited about? Can you share some stories or just talk about the work that you're doing together?

00:00:59.56 >> So I guess I'll jump in and say something that is kind of exciting that we're working through is the EMIS work group for special education, where we're sitting down and talking to the field and getting their feedback in regards to particular related services and whether we think that those would be program codes that would be worthy of having data reported on from the field. I'd definitely be part of that process newly and kind of engaging in that conversation. I definitely think it's exciting because it's just another extension of why it's very important that we have open perspective and collaboration not just within the data team but across the department and even with the field. Having that input from that work group, for example, has really provided a lot of insight about ongoing concerns or things that maybe we may be able to implement changes or processes of improvement, and I definitely think being a part of that new process is exciting and definitely puts our minds in the output of thinking, how can we provide better processes, whether that's coming from the data manager's side of looking at the data and analyzing it differently and maybe making adjustments to our code for the indicators, or maybe if it comes to collecting new and more important data as it is with related services and then also thinking, how can the data team and the monitoring team all come together to collaborate with us on figuring out what are new processes that we can put in place to really help bring forth any issues that we're seeing from the field and be able to address it appropriately. So I definitely think that's an exciting initiative that we're working on, and it really drives home the importance of us all working together cohesively. So that's one thing we're doing.

00:02:45.30 >> Another thing that we're doing is working on getting more of our special ed data out to the public. So a lot of our data is buried within our profiles, and you can find it, but it's not in a place where most of our internal and external stakeholders know to locate it. So within the data quality and governance office, we are working on multiple Power BI reports but on the same page where our local report card is, and we have a page called Advanced Reports where there are just tons and tons of public data available for both internal and external stakeholders. So it's a slow-going process, kind of a painful process, but by going through that theme, it'll make the special ed data more visual-able, and I think by doing that, it'll make people pay attention to special ed data more instead of just thinking it's kind of a side thought.

00:03:50.77 >> Yeah.

00:03:51.00 >> So ...

00:03:51.77 >> And you all have such an amazing significant disproportionality dashboard, too. So I'm sure you're kind of maybe using that as a model in some ways, or I just always really turn to that example from you all when other states are trying to do more of that work on getting their data more publicly available. So I'm sure what you're working on now will be fantastic, too.

00:04:18.72 >> I'm glad you brought up significant disproportionality because that's one of the things I was going to say that we've been excited about, that we recently did a bunch of changes for kind of in the interest of getting more data out around significant disproportionality even just through our special education profiles and the display that we use for the data. We've changed a ton of those display rules over the last ... I want to say two iterations reporting, so that we're showing now all of the data that we have for significant ... or disproportionality whereas previously we were only showing data for the districts and community schools who met the minimum group size. And if you're familiar with significant disproportionality, you can see how that might be problematic for districts who could then potentially be surprised that they've been flagged with significant disproportionality if they didn't get a warning in their previous profile that maybe the first year they didn't meet the minimum group size, and then the second and third year they were above that state threshold, and now the next profile they get flagged, but they didn't have that warning previously, so they didn't really know it was coming. Not that we had that happen at all, but in avoiding that now is that everybody can see whatever data is reported, even if they're not meeting that minimum group size. And then we've also added as kind of ... As part of our monitoring process, we provide action statements that they know of the processes they need to go through if they do hit on any of our indicators, and so one of those that we've added for disproportionality is a statement that tells them that they're at risk. So not only can they see all of that data, but they're getting an explicit language there that says, "Hey, you're at risk if this ... If you have a threshold above 2.5 for a third year, you'll be flagged in our next profile as well."

00:06:14.00 >> Hmm.

00:06:14.22 >> Just to jump in and add a little bit to that, another thing that we do, kind of focus on is continuous improvement of that monitoring tool that you had mentioned, the ... our special education profile. So in addition to those dispro things, we also have brought in some additional support on our indicators about graduation and dropout. So we've added those to the process as far as supporting districts and having required actions depending on their data in those areas as well. So it's really nice because we always do have an opportunity to take a continuous improvement mindset when it comes to the special education profiles and think about, "Okay, what are state initiatives? What do we really need to change our focus on in the office?" The reason that we kind of brought those into our special education profile monitoring system in the first place is to address our areas for improvement in our state determination. So it's really nice to kind of have that connection and to have that feedback and to be able to do those continuous improvement changes as well as supporting what we already have in the system.

00:07:19.75 >> Great. Well, there's certainly a lot happening of a lot of exciting work going on. And what about for the future? What do you have coming up that you'd like to highlight?

00:07:31.95 >> Well, based on what Matthew just said with our exiting indicators, our graduation and dropout indicators, we are looking at potentially changing some of our business rules for those in terms of how that data is being reported because we're getting a lot of feedback from the field having this been the first time that we've had required actions on these two indicators, where we're getting a lot of feedback from the field. We're processing a lot of that internally. We're getting a lot of feedback internally as well. So we're considering some updates to those indicators going forward which I think will better our reporting in the future, so changes both in our internal processes but also improving our products in the end.

00:08:14.41 >> Hmm.

00:08:14.54 >> Data, data, data, so glad I remembered. One of the things that excites me and what we have coming up in the future, too, is then the compilation of all the data that we can share with many of our stakeholders, districts, individual, our regional systems of support and statewide because if we can provide, "Here's our overall profile as a state. Here's where our strengths are. Here's our opportunities for improvement," and I think getting that out to people and especially some of our stakeholders like our educational service centers and state support teams to help us help the state and districts move our numbers is pretty exciting. And let's face it, who doesn't get excited over data? Come on.

00:09:00.77 >> I think all of us do at least.

00:09:03.33 >> I was going to build on something that Ashley said and also refer back to what's a challenge and what's an advantage on being on the smaller team is that through monitoring, one of the data things that we collect is through the review of the evaluations for indicators nine and ten. We look at five records per disability category for which the district is flagged, and the beautiful thing about that is there are two of us that do it, so we're really great iterators, but also there's just two of us that do it. So that month of January so that we can collect all of that and give it to Matthew, who does our APR, to get that done, it's a challenge, but that's where we get a lot of information that we're able to share out to the systems of support for things that are needed.

00:09:50.50 >> I'm really proud of the work that our team does, and I just want to read to you the vision statement that the team came up with. This is part of our own strategic planning. The vision of the data team is to articulate the experiences of Ohio students with disabilities by using data that improves educational outcomes and learning experiences.

00:10:15.92 >> Nice.

00:10:16.98 >> Yeah, I think so, too. I think so, too.

00:10:19.54 >> Even just having a vision statement, I think, is pretty cool.

00:10:23.59 >> Our mission statement is a little longer, but the vision really sets us out or where we ... That's where we see ourselves headed.

00:10:31.70 >> Yeah. Well, thank you all so much. This was really informative and so interesting to hear about the work you're doing, especially how you're working together as a team. That's, I think, just so exciting and unique.

00:10:49.26 >> This is Johanna. So thank you for inviting us and our team to come on air.

00:10:55.71 >> Mm-hmm.

00:10:56.17 >> Having a team is really a special thing, but they are excellent what they do. They do communicate at all levels and all different means of communication. They not only are sharing and communicating things out, but I think you heard them say, too, they're excellent at listening. So they're listening to stakeholders. They're listening to our regional support teams. They're listening to districts, and they're bringing that information back to information how we can improve our process over and over and over again, and to have a team that's dedicated to doing that is, as a state leader with this work, is amazing. Just there's so many different angles and so many different perspectives that come together to drive that. It just compounds. It multiplies faster that way, and they produce work that's amazing. So I'm very thankful for them, as a team, as well.

00:11:51.06 >> Well, thank you all again for your time and just sharing how this has been working for you all, and it just is so exciting to hear about, so thank you.

00:12:04.67 >> Thank you, Amy, for having us.

00:12:08.29 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.51 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.46 >> Hello, thank you for listening to "A Date with Data." We are doing something a little different today, and I have Ohio's whole IDEA Data team joining to share how they are working together to collect, analyze, report, use and improve their state's IDEA data quality. I'd like to introduce our guests from the Ohio Department of Education. We have Jo Hanna Ward, who is the Director, and Joe Petrarca. I would like to introduce our guests from the Ohio Department of Education. We have Jo Hanna Ward, who is the Director, and Joe Petrarca, who is the Associate Director in the Office of Exceptional Children. Also from the Office of Exceptional Children, we have Matthew Loesch, Ashley Rector, Shauna Schramke and Debra Shirley, who are all Education Program Specialists, and also Kaellen Craft and De'Ja Roundtree-Gibbs, who are Data Administration Managers in the office of Data Quality and Governance. So welcome to all of you. I wanted just to start things off by asking you to tell us how you work together as a data team. What does that look like?

00:01:35.71 >> Well, I'll go ahead and start. I'm Joe, one of the associate directors in our office. Then I'm going to push this off to the team for them to finish answering. So the team works together very closely. This is one of the closest teams I've ever worked with, and part of that commitment that we have to providing quality and excellent data is constant communication, whether that's virtual, in person when we get together, email, chat. We do all of that, and that's how we are able to move our process, procedures and our product out the door. Who wants to follow up with that?

00:02:17.74 >> I can, Joe. So this is Kaellen Craft, one of the data managers who works very closely with our wonderful data team. So I think our setup is very unique, that we have a set of education program specialists that work very closely with the data managers, and I think it's a really interesting setup because we definitely bounce ideas, woes, a lot of woes [Indistinct], work, challenges off of one another. It's not as though we are really siloed. I think our silo comes just by the nature of the work that we do, whether we're pulling data for our program office and then the EPSes are analyzing and synthesizing and putting the data together, but we still work very closely together. There's a lot of collaboration on all of the products and services that we provide.

00:03:17.62 >> This is Ashley. I'll build a little bit on what Kaellen shared. We are a fairly young team. We've only been around for about 7 years. However long Matthew and I have been in the department, that's about how long our team has existed. And we started out with just the three of us, Matthew, me and a supervisor and one data manager at that time, and we have grown now to have, gosh, I want to say three data managers on the special education side and one on the gifted side. And then four of us on the data team within the program office. That's Matthew and I kind of on the data, the more technical, methodological side with the calculations, and then Debra and Shauna are both kind of spearheading all of the monitoring of those indicators.

00:04:09.84 >> Great. Yeah, you have a really great structure, it sounds like. I think that's a big reason we were hoping to have you all, of course, on the podcast to talk about this kind of unique way of handling your state's IDEA data, where often we see one data manager in a state, maybe there's two, and that's their role in totality. And others maybe don't know a whole lot about what they're doing and vice versa, so it's just a really unique setup, and it sounds like there's a lot of great collaboration going on that really helps drive the quality of your data that comes out.

00:04:50.88 >> It really does. And in our office, we're really large. We're the largest office at the department, and we have ... Alone in our office, we have seven sections, of which our data team is one of, and we make those purposeful connections with each of our sections because the data drives everything that they do, in each of those sections. And one of our sections that we work closely with is our Urban Support team, and they provide support to 11 urban districts in Ohio. And the data that this team calculates and puts out is important in helping our urban districts move forward with their initiatives as well as the initiatives that we have with them. So it's ... And so we make those purposeful connections not only with our office, but among the department as well, so sharing that data with our Preschool office, Office for Early Learning and School Readiness as well as our offices that we connect with on a regular basis, like our Graduation office. And of course, then, the connections that we make all of our stakeholders in Ohio, which are many.

00:06:10.40 >> Mm-hmm. Yeah, it seems like it's a culture within your office of collaboration and really kind of built into the system, so that's wonderful to hear.

00:06:22.28 >> Yeah, it is, and that's one of our values as a team, is ... and we do ... We did some mini strategic planning not too long ago, and that was one of the values that we explicitly stated was collaboration.

00:06:37.38 >> It's very intentional about it, which I think makes a big difference.

00:06:42.12 >> Yes. Yes, and intentional with each other as well. Nobody is kept out of the loop on anything.

00:06:48.82 >> Mm-hmm. Mm-hmm.

00:06:49.89 >> We believe in shared leadership in our office, and that's set by the example made by our Director, Jo Hanna. And so it's not just us making the decisions. We make the decisions as a team, and yes, there are some that land with the team, some that land with me, some that land with Jo Hanna, but we're all involved in that decision-making at some point or another.

00:07:15.46 >> Hmm. Yeah, I love hearing that. I'm going to come work for you guys, I think. What are some of the real advantages that you have found to working together as a team, and are there any challenges too?

00:07:28.65 >> This is Matthew. I'll jump in on some of the advantages. So something that I really love about our team, and I think is kind of unique about the way it's structured, is that we have the opportunity to get the perspective of somebody working with data at all different points. So we have our data managers, that they're really very knowledgeable about how the data is collected and how it's stored and what we can get from the system and what we can't. And then the data specialists on our team, we kind of know how it's ... how the data's typically used by the program office, and often, maybe we know the best way to kind of explain and communicate the data throughout the office, knowing the caveats that would probably be important to include and any kind of ... just any kind of things that we might want to make other people in our office aware of before they're communicating with the districts. And then our district support specialists, they kind of know the common questions and pitfalls from the districts, and so they can kind of take a look at that data from that perspective and thinking about how are districts often being tripped up by this. Is there anything that we can clear up and make a little bit more streamlined so that the districts can understand it better and then they can have an easier time using the data as well. And another thing I would say is a big advantage in our team is that we get lots of opportunities for peer review. We can have multiple people, multiple eyes taking a look at the data for accuracy, which certainly helps to make sure that we have the most accurate products going out the door, but I also think it really helps as far as having a lot of people with different perspectives can really help with coming up for ideas for improvement, thinking about new processes, new approaches, new safeguards and just any kind of new way to use the data. It's really helpful to get all different perspectives on that.

00:09:16.82 >> Great point.

00:09:18.77 >> And I guess I could say, just as far as challenges goes, we've kind of already touched on it a little bit, but certainly more communication is needed. We always have to make sure everybody's at the table, and we really want to make sure that we don't have anybody feeling like they don't at least get input and have the opportunity to weigh in because it's so important, just because we all have such different bodies of knowledge and different specialties as far as the data and using it and caveats and all of that. So it definitely can be more challenging with scheduling and communication, just making sure that everybody's included in the loop, that we get everybody at the table and looking at everything and not leaving people out in the cold, for example.

00:10:02.50 >> Mm-hmm.

00:10:03.92 >> I can add on with a challenge. We do have a lot of data needs across the office, outside of just our federal reporting requirements, and there are ... We sound like we're a large team, but there really are only, at most, two people at each point in the data process that are working on each point. So we have our data managers, Kaellen and De'Ja, who are primarily working on our special education indicators. We have an early learning data manager and a gifted data manager, who are kind of working on other parts. And then Matthew and I are the only ones that are touching kind of the formatting of the data quality checks and the methodology between the data managers and what you see as the final product. And then only Debra and Shauna are monitoring all of those indicators for all of the districts in the state, so I would say that, I guess, it kind of has its pros and its cons, right? It's good because it's easier to get people at the table. It's easier to have a smaller team with communication, so that structure is good in that way. And then, I guess on the other side of that, is that there are only two of us, so if one of us is on vacation, then everything else in that process falls to that other person.

00:11:18.79 >> Mm-hmm.

00:11:20.58 >> Which is why no one's allowed to go on vacation.

00:11:24.76 >> Sorry.

00:11:25.82 >> Yeah. I think we've done a lot of cross-training, and that helps, but again, that's a strength with this team. Everybody's willing to step in and step up when we need to, and yet, again, it's a small team that, in the end, is going to have over 4 million data hits. That's a lot for such a small team to do, and plus, it's a lot to monitor those with basically two of our team members. Another challenge is we collaborate and communicate across the department and the office. It's kind of keeping and making sure that our data is front and center to the other places that it needs to be within the department. And it's a good thing, and it's also a challenge as well. And that's ... Part of it is timing. We're all busy. Each office is busy, and we're a state agency, a large one, in a large state. And everybody has their own initiatives, so it's making sure, too ... I think another challenge is making sure that the data is connected in meaningful ways to everybody so that they can see it, so that they can see that by looking at the data ... and I'm doing air quotes ... just for students with special needs impacts everything. It impacts their achievement. It impacts the districts' overall achievement. It impacts our state determinations as well, and we're trying to make it so it's not just something in isolation.

00:13:04.65 >> Yes, for sure. It needs to be a full Department of Education priority and discussion, not just isolated within ... yeah ... special education. And it sounds like you all are very aware of some of the challenges with having this kind of team approach, but you really have struck a nice balance of that cross-training piece of it and then still having, though, those different perspectives, like Matt was talking about. So kind of having both backups and also maybe folks who aren't quite as immersed in the day-to-day but can be a sounding board and brainstorm and give feedback and have some idea of kind of what's happening in this other area that maybe they're not as immersed in.

00:13:52.45 >> Yeah, it really helps just given the diversity of our team. So Matt and Ashley are really experts in looking at the data and spreadsheets and compiling all of that, and Debra and Shauna's expertise comes in from a classroom perspective. And as we begin looking at perhaps what could be root causes of things or helping districts with the monitoring and development of any of their plans, that's where their expertise comes in. And it's really been helpful, I think, to have those different perspectives. What also is helpful is that we have a director who continually beats the drum about the students that we serve in our office and their achievement and how important that is, and that they are important ... It's important because they are all of our students, not just students with special needs first. And I think that's a great concept for educators to have.

00:14:52.57 >> I want to build on what Joe said, is when Shauna and I are monitoring Joe and Jo Hanna, leadership has really given us the permission to move forward and see what we've learned through monitoring, so that we can identify trends across the state and how we can use that and how we can do better and working with our state support teams within their regions to see, do we have pockets of things that can be addressed to help ...

00:15:16.62 >> Thank you for listening to part one of my conversation with the Ohio data team. Please join us on the next episode for part two of our discussion.

00:15:27.09 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.51 >> Welcome to our new episode of "A Date with Data." Today, we are talking to Cindy Brown, who is the 619 Coordinator, and Bob Sabolcik, who is the Part B Data Manager. Both are with the Delaware Department of Education, and they're going to be discussing their Preschool Least Restrictive Environment, or Indicator 6, journey. Can each of you just say a little bit about yourselves and your role, and, Cindy, do you want to kick us off?

00:00:54.44 >> Yes, thank you, Amy and IDC, for having us.

00:00:57.66 >> Thank you.

00:00:58.18 >> I am about ... Thank you. I'm about to finish my 7th year here at the Delaware Department of Education. Prior to that, I was in Maine for many years as the 619 and Part C Coordinator. And I really am very passionate about this work and about making sure that we're expanding inclusive opportunities for our preschoolers here in Delaware and really appreciate the opportunity to work with our Part B Data Manager on this.

00:01:27.98 >> Thank you. Bob?

00:01:30.73 >> I am Bob Sabolcik. I have been with the Technology and Data Office of the Department of Education for a year and a half now. Before that, I was in the special education classroom for 23 years in the secondary setting, so I bring kind of an interesting perspective that most data managers don't have from the special ed background, and it's been a good blend so far.

00:01:56.16 >> Yeah, absolutely. I love asking that question because I always learn completely new things about state folks and their experience and background, so thank you for sharing.

00:02:08.84 >> Absolutely.

00:02:10.55 >> So to get us started, can you just tell us the story of your Indicator 6 data? What are some of the data quality changes, maybe, that you've encountered, and also, what are some of the successes and things that you think you're really doing well and are excited about?

00:02:28.32 >> Yeah, I can jump in, and then I'll kick it over to Bob. Yeah, as I said, I'm very passionate about this and had the opportunity to work with another TA center on the Expanding Inclusive Opportunities Initiative. And so when I reviewed in Delaware and looked at our data, we were very far away from meeting our targets at that time. And as I got to know our LEAs ... I meet monthly with all of our local 619 coordinators, and hearing from them the various challenges that they've been experiencing, it really heightened my enthusiasm about really trying to change the trajectory on the path we were on here. And one of the first things we turned to was starting to really look at the data and dig into it a bit and kind of walk through people, what the definitions were and ... for the different settings. And found out that a lot of the districts were not interpreting those placement codes correctly, so a lot of children were being not coded correctly. And then coupled with, at that time, kindergartners were still included in the count, so I did have the opportunity to work with both NASDSE and OSEP to kind of put forth the effort to change how the count is done and taking those kindergartners out of the Part ... this Indicator 6 count. And we knew that that would make a huge difference, right?

00:04:19.61 >> Yeah.

00:04:19.82 >> By and large, at least in Delaware, kindergarten children were in inclusive settings or in those A settings, and so that really peeled back a layer of the onion and let us see that we had even more challenges. So getting people to understand how to code the data has been a major emphasis. And again, we really invest time in looking at the data and talking about it on a very regular basis. And, Bob, I don't know if you want to add to that in terms of any other data quality challenges that you've seen since you've come on.

00:05:00.06 >> Well, I think Cindy hit the nail on the head about the idea is, if the information isn't entered correctly, the settings isn't entered correctly, we're not really going to get true accurate reporting. And really is ... There is a level of complexity to how a case manager is going to figure out like, "Well, what setting I should pick of all these settings based on what the district or the school has available?" So it can be complicated and especially when you look at our data collection tool. It is based on a PowerSchool special ed screen where they enter their data, and it's not necessarily explicit enough where it gives that second check of the case manager, are they really kind of doing it right? And I think that is one of the major challenges from the data standpoint that we're seeing.

00:05:54.97 >> Yeah, I agree, Bob. That's good point.

00:05:59.15 >> And to kind of ... It sounds like you've put in place these monthly meetings, which is great, with the local 619 coordinators. And what are some other things that you've done to kind of help address that issue of the coding piece?

00:06:17.96 >> Well, when the first B6 calculator came out, we immediately turned to use that because that really did break down and drill down the data and really looking at the districts. And we gave all of the districts a copy of everybody's data. A colleague of mine used to say, "Sunshine is the best disinfectant," and so really looking at ... And again, we're a very small state, right? Three counties. Everybody knows everybody or knows somebody that knows somebody, right? But really looking at that and then trying to encourage districts to talk to one another. Some districts seem to be doing fairly well with Indicator 6A and having those inclusive classrooms. Others really, really were struggling, so encouraging districts to talk to one another and strategize. We've also spent a lot of time on the preschool finance tool kit and looking at, what are some ways that you can expand those settings? We don't have universally-funded preschool here in Delaware. Our ... We do have 4-year-old slots funded, but it's a very, very small amount.

00:07:40.31 >> Mm-hmm.

00:07:41.39 >> So trying to look at those integrated opportunities has been challenging. Most of our districts resort to using their classrooms that are located on their district campuses, so also looking at expanding the use of the itinerant model and also trying to partner with community programs, especially Head Start, and private childcare centers as placements or as options for children services. The itinerant model has expanded enormously since I've been here. We now have it spread throughout the state. We have a community of practice, and we've seen some districts who now are in double digits with itinerant teachers and not just special ed teachers, but also sending out the related services itinerantly. So that model really has expanded, and we're trying to refine that further to really look at using other evidence-based practices, like coaching and collaborative classroom consultation, as a way to deliver services to the children and the other adults who are caring for them during the day.

00:08:55.91 >> Great, so all is ways ... All of those sound like ways to kind of increase inclusion so that there's more students with disabilities in those kind of more general ed settings. Those are good strategies.

00:09:07.36 >> And we have created an inclusion guide that has been published on our website. We've given out hard copies to not only our districts, but our community-based programs to really emphasize access, participation for all children.

00:09:26.80 >> How have you been creating a culture of data use in Delaware around Indicator 6 with the districts and other stakeholders?

00:09:38.68 >> That's a great question. We talk about creating a culture of data use and compliance. And so at our monthly 619 meetings, or early child special ed meetings, we've reframed it, we now have switched to taking the last half hour. That's actually a professional learning experience, so they do get recertification credits for that. It's a full day meeting once a month, but we've taken that last half hour of the agenda now and given it back to them to say that, now, this is your time to go work on your data, to either make sure you've got everything entered, accurate, timely, correct or to review data or to do whatever you need to do around making sure that you're having that conversation with your own internal team around the importance and value of, as Bob said earlier, collecting it correctly, getting it entered in a timely way and then using it on the back end.

00:10:41.81 >> Yeah, that's really an interesting way to approach it because you're kind of blocking out time for them in their calendars because that's all we need more of, is time to do all these things.

00:10:50.75 >> Exactly. It's already designated time to them, and so we're just saying, "We're going to stop our formal meeting a little earlier and give you this block of time, but we expect you to use it to really work on your data issues."

00:11:04.49 >> Mm-hmm. Great. And we pretty recently hosted ... IDC did, a Hands-On Learning Academy, or HOLA. We've done a few of those, kind of focused on different tools or resources that we're developing and bringing in a small number of states to really test them out and use them, plugging in their own data and kind of work in smaller groups to learn really how the tool works and how they can use it back in their own states and with their own districts. And I know that you all participated in one that we did on the Preschool Environments Tool Kit and Calculator, so can you just talk about what that experience was like. What did you take away from it? Anything you learned that you're going to apply now?

00:11:55.34 >> I'm going to give a brief answer and then let Bob talk about this one. Just in brief, it was great. It was a huge success. It was probably one of the most worthwhile things that I've gone to in a very long time.

00:12:09.33 >> Great.

00:12:09.84 >> So, Bob, I'll turn it over to you.

00:12:12.35 >> Yeah, from the two aspects of that, being at the meeting and being around the other managers and data coordinators, is a really valuable experience, I think, because you get so much more out of it from being ... from having in person and really talking to others in different states and how they interact with their LEAs and interact with their data. And also, I was able to really learn a lot. Even though I have 20 years more experience in special education, it's in a secondary level.

00:12:45.24 >> Mm-hmm.

00:12:46.12 >> I really needed to learn a lot about what's going on at the pre-K level, and it's a different placement sets for that. So that was a great learning environment that I was able to learn from Cindy and from the other folks there. And the tool kit is great because it's a nice standard tool kit that everybody can use, and it's not something that I had to create, which is nice. And it really works well. You just plop in your data. You have pivot tables. You have all the different tools that I could create on my own, but I don't have to, and then other people can use it relatively easy. It is a fairly complicated tool, but it's not ... It's set up in a way that you don't have to have a high degree of technical knowledge around pivot tables and data to be able to use it to get out meaningful information.

00:13:44.07 >> Yeah, I think that's a great point, Bob, and I think we appreciate ... It was six states total that were there, and everyone brought their data manager, which was great, because sometimes we do have these 619 experiences, right, but it's usually a state coordinator. It's not always the data manager, or we're not always there together. So having it in that smaller forum that was ... And it was 2 full days that were really concentrated. The discussions that happened were fantastic. There was time allowed for deeper dives and more meaningful interactions, and Bob was a superstar. He had his computer out the whole time, and he was working on data. And everybody got really into it, and so if folks get an opportunity to go to one of these, I would say, "Absolutely, jump at it."

00:14:39.82 >> Yeah, definitely.

00:14:40.43 >> It was really valuable. Yeah.

00:14:43.74 >> Great. Well, glad to hear you got so much out of it, and it was a success. So we definitely ... Yeah, we'll be doing more of those.

00:14:52.74 >> Definitely.

00:14:53.04 >> What do you ... Yeah.

00:14:54.44 >> I was just going to say kudos to ... it was Carol, Carol Seay, [Indistinct] and Tony Ruggiero, and they just did a fabulous job of organizing it with the content and, again, allowing time for meaningful interaction amongst the different state folk and allowing us to explore and ask questions of each other. And it was just invaluable.

00:15:21.84 >> Wonderful. So what do you have coming up? What are your plans for continuing to push and make improvements in the quality of your Indicator 6 data?

00:15:34.10 >> We have big plans. So one of the biggies coming right up is that, for the first time in Delaware ... And I don't know about a lot of other states that are doing it from what I hear. We will be including Indicator B6 in the local determination, and so if the LEA is not meeting the targets, they will have to do a continuous improvement plan. And really, it's the first time we're really holding them accountable for looking at strategies and innovations to how to move the needle on their data. And with this last creation of the new SPP, we had robust and many stakeholder meetings that were facilitated by IDC, and we did option to go with the discrete ages. So now we have targets for 3s, 4s and 5, and that also was very telling, and with this new APR, that one of our biggest challenges is finding those placements for 3-year-olds that are inclusive. And so now that we're going to put this out next month of the local determinations, in turn, they'll be able to see where their pain points might be or more painful points might be and can allow them to zero in and really think about if it is 3-year-olds that are challenging, to zero in on that. And the other thing is, we will be ... Once we get this child count submitted, we'll be putting out the Preschool Calculator tool, giving all the districts their individual data. And one of the big improvements to the tool now is it's one sheet, so you can see your data, your actual numbers and the percentages in one place, that used to be two separate reports. So seeing that, I think, is going to be more impactful to the districts as they look at that, look at their local determination and plan for how they can really start to move the needle.

00:17:50.00 >> Wow. Well, it definitely sounds you have some good plans in place and excited to hear how things go with including Indicator 6 in your local determinations for the first time and, yeah, really putting some teeth in it and see if that can help, like you said, move the needle. Great. Is there anything else either of you want to share that you didn't already mention regarding Indicator 6 and data quality?

00:18:18.95 >> One other thing that came out of the meeting, while Cindy and I were talking and reviewing the documents provided by IDC and our own data collection tool, the IAP, is that we're looking at improving the IAP because whenever I was looking at the decision tree for preschool education environments, I thought it was a great flow chart where you go yes or no.

00:18:42.54 >> Mm-hmm.

00:18:42.75 >> And I thought that would be a great way to modify the IAP so when ... instead of just looking at a list of the placements, that the case manager would kind of be guided through the decision tree, and then they would get to the placement, just to give them that next layer of, "are you really putting the student in the right placement?"

00:19:06.75 >> Hmm. That's a great idea.

00:19:08.56 >> Yeah, that's an excellent point, Bob. We are talking about revising our ... We do have a preschool version of the IAP here, and we are looking at revising that preschool IAP and really changing up how the LRE piece is displayed and talked about. Yeah.

00:19:26.76 >> Hmm. Yeah, I don't know if I've heard that really before, so that's another really great innovation and way to kind of use tools and things that are already exist, like that decision tree, and kind of in a different way than maybe we would've thought of before, but to really help guide everyone to the most accurate placement code.

00:19:50.83 >> Right now, we're in a feasibility stage because we're a little bit at the mercy of PowerSchool IAP and what they can technically do on their end for us, but we are in the middle of an FRP, so we'll have to see whoever wins the contract for our student information system to see if that's something that we can talk to them about.

00:20:12.18 >> Hmm. Great.

00:20:13.76 >> I think our mantra too, at this point, is status quo is not an option. We really ... And, Amy, we talk about this on a national level, right?

00:20:29.04 >> Yeah.

00:20:29.77 >> We haven't moved the needle on preschool LRE in 40 years. OSEP is aware of that, and so we really have to be thinking about innovations, creativity, public/private partnerships. I refer, frequently and often, to the "Dear Colleague" letter of 2017, reminding people about that ...

00:20:58.21 >> Mm-hmm.

00:20:59.20 >> ... and that we really do have to ... We can't just sit back and admire the problem anymore.

00:21:05.76 >> Yeah.

00:21:06.09 >> We really do need to take some affirmative action to change things up.

00:21:12.91 >> Yeah, and it sounds like you all are really doing that, so kudos to you.

00:21:17.45 >> We're trying. We're trying.

00:21:19.32 >> You are. All right. Well, thank you both so much. Really appreciate you being on and telling us about your experiences at HOLA and just what you've been doing to really try to improve your Indicator 6 data, so thank you.

00:21:35.51 >> Thank you, Amy.

00:21:36.83 >> You're welcome.

00:21:38.96 >> To access podcast resources, submit questions related to today's episode or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###
00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:25.00 >> Welcome to part two of "A Date with Data" with North Carolina's Kelley Blas and Lauren Holahan. If you missed part one, please make sure to check it out, and in this episode we're going to continue our conversation on how North Carolina has kept stakeholders meaningfully engaged in their SPP/APR.

00:00:45.40 So through all of this, what are some of the challenges that you've faced in getting stakeholders to attend? It sounds like you had maybe not that much of an issue with that. You had a lot of folks, but keeping them engaged and continuing to have them be part of this work.

00:01:07.60 >> How long have we got for this one?

00:01:14.43 >> Well, I'll just start with the first one, and then again pass it over to Lauren because there is quite a few challenges that we faced, but I would say our main one ... and I think most states would probably agree with this ... is parent involvement. A success in that is that, as Lauren mentioned earlier, we did have multiple meetings, and we scheduled the second meeting for each target setting group in the evenings. So it was a 6 o'clock meeting. That is where we had our most engaged parents, and they did ask lots of different questions that only parents would come up with, and their perspective was totally different than what we would find on our morning meetings, so that was a good thing. We just didn't feel like we had as many as we would have liked and as diverse as we would have liked as far as racially diverse, students with disabilities, the different disability categories. We would have liked to have seen more parents, and we're still working on that with our Parent Advisory Council and our other parent advocacy groups on involving more parents. And, Lauren, what do you think?

00:02:27.48 >> Yeah, yeah. I think another challenge is sustaining the effort and the momentum after we got the targets and the baselines selected, just sustaining the engagement because the conversation with stakeholders changes after that, right? After ...

00:02:46.22 >> Yep, absolutely.

00:02:47.24 >> ... that work is done, we have new SPP. Awesome. We ... But I think just keeping that momentum up especially for those stakeholders who aren't involved in a work group, so we've been talking. Kelley and I and our team have been talking about how to kind of recharge that external stakeholder group and ways to ... for us to kind of plan a little bit further out in terms of talking about progress, talking about ... thinking about some of the targets we potentially need to reset. Kelley, I wonder if you want to talk about just some of what we're experiencing in terms of multiple initiatives and leadership changes and ...

00:03:32.04 >> Oh, my goodness, yes. So along with having this huge task of a changing measurement table and getting this information out to our stakeholders and having them work through that with us, and I'm sure many states are in the same boat, but we also had a lot of initiatives going on within the department. Of course, we decided to change our SIMR as well, and so that was a huge change. We went from focusing on graduation rate to focusing really down on our fourth-grade reading and the opportunity gaps between students with disabilities and non-disabled students and between students with disabilities of color and students with disabilities who are white on our State-Identified Measurable Results. So that change alone is a huge shift in opportunity to engage all of our stakeholders, all of our districts in understanding what that looks like and the engagement that they need to have there. And we also did some major changes with our indicator 4, 9 and 10 process as far as reviewing and revising their policies, practices and procedures and what that looks like. And then there were some other division priorities that the PSUs needed to engage in that were very large compliance-related things that they had to roll out, so there was just so much happening in the past 2 years that there's ... I know that there are times, and they have told us directly that there are times when they just feel overwhelmed by the amount of things that are happening and changing when it comes to data and priorities. We also had significant staffing changes, lots and lots of long-term department folks have retired, and so training new staff and helping them understand what our priorities are and getting everybody onboard has been a huge challenge for us, and I'm sure, like I said, I think many other states are experiencing the staffing changes at least, if they're not experiencing all these other changes which some ... some of which we brought on ourselves, Lauren and I.

00:05:52.24 >> Yeah.

00:05:53.53 >> But you had long-term aspirations in mind. It's all working towards improvement and better outcomes so ...

00:06:01.62 >> I think one of the things that ... especially as we've done some revision to how we think about an approach and address indicators 4, 9 and 10 and significant disproportionality, I think what we've experienced is quite a bit of isolation by our local directors of special education in their context, and those indicators and significant disproportionality, those data, we have a friend in an LEA that says that general education is writing the check the EC has to back, right?

00:06:36.33 >> Mm-hmm, mm-hmm.

00:06:37.13 >> That it's general education policies and practices that are driving those particular indicators and significant disproportionality data, and yet the understanding of those requirements, what those data even mean on the general education side is pretty low, and so I think we've got a significant opportunity to do some capacity building with our gen-ed stakeholders at the local level.

00:07:08.64 >> Yeah, for sure, I think that's a common theme and something that I know, in the special ed world at least, trying to continue to push harder to get general ed involved in all of it but certainly those indicators. They really need to be at the table and understanding and part of all of those conversations and decision-making.

00:07:33.53 >> Kelley, will you just talk about sort of the balancing act, I think the challenge of keeping the SPP/APR the right size?

00:07:42.11 >> Well ...

00:07:42.42 >> For us data people ...

00:07:45.21 >> Right.

00:07:45.94 >> ... there's a cliff that we often run up to and like, "Whoa!"

00:07:49.96 >> Right. So, yeah, I know that for me specifically, I do focus on the end game and the targets and the outcomes, but it is important, and I think we've learned, especially through all this work and part of that challenge of all of the different things that we've been implementing, is that we're learning how each one of those indicators actually feed into the bigger picture of significant disproportionality of the policies, practices and procedures that are happening in districts, and so getting down to the student stories that the data represents. And Rachel Wilkinson did a wonderful job of showing us the journey of the indicators on a student level, and so we are really trying to help our districts understand that all of these play a part together and that the outcome or the percentage that they're at at the end is not as important as ensuring that their meeting the needs of each and every student.

00:09:02.79 >> And through all of those kind of challenges that you talked about, it still sounds like there were a lot of strategies and lessons learned and ways that you dealt with those challenges and really faced them. Can you share a few of those? Are there other strategies that you have found to really help keep stakeholders engaged that you can talk about?

00:09:26.52 >> So other than the work that we've already been doing with our stakeholders around the work groups, we actually have been planning to implement strategies to engage, and one of those is going back to that year-long plan or road map for discussing our indicators. So our intention is that we'll have an indicator highlight at each one of our stakeholder meetings and so that ... Again, so that we're not overwhelming them with, "Here's the progress on 17 indicators." We will highlight one or two indicators and just talk about it well, as talk about considering resetting some targets where we know that, based on the data and the timing of those changes, that OSEP [Indistinct] with the measurement table that we just totally missed the mark as far as us hitting those targets.

00:10:18.50 >> Well, I think ...

00:10:21.01 >> Example: indicator 3D.

00:10:22.91 >> Yeah.

00:10:23.24 >> Right.

00:10:23.36 >> Yes, but it was new and to ... I think probably all states missed the mark, and just you don't know what you don't know. It's going to take a few years, as we've seen, to really understand and get what's happening and where you want to go.

00:10:40.78 >> Yeah, so I guess what we learned in that was, if you don't know and it's a new measurement, choose conservatively.

00:10:52.03 >> Right. Yep.

00:10:54.40 >> Yeah, yeah. I think the other thing ... and we've mentioned it before, but even as we've kind of retooled and revisioned the way we're doing our local self-assessment for indicator 17 around fourth-grade reading outcomes and have sort of chunked up the work of that self-assessment into different phases this time, our LEAs just finished a data analysis, and as they move into root-cause analysis at the local level, supporting them, sharing their data analysis with their local stakeholders, what we're finding is we're just continuing to learn that there really is an art to the dosage of data that you share with any stakeholder group at any one time and just to be strategic about which data and how much and how much space to leave for people to respond to it. I think we're just learning, even ourselves and as we're coaching local implementers that it's ... When you're talking about data, it's super easy just to get on a monologue and just share all kinds of charts and graphs and then run out of time, and all you've done at that point is overwhelmed people.

00:12:15.00 >> Mm-hmm.

00:12:15.35 >> And to truly engage stakeholders, like that dosage thing we're still learning.

00:12:22.76 >> Yeah, definitely an art, and when you figure it out, let us know because it is. It's so true. Just knowing your audience as much as you can, and sometimes you really don't. If you're kind of putting a call out and parents and ... It's a whole range of people that have different levels of understanding and interest. It's definitely a challenge, but it sounds like you're really working hard to figure out what that mix is and what works, so I applaud you for that. You talked a little about upcoming plans already. Anything else that you haven't mentioned that you want to highlight?

00:13:01.85 >> Well, just before we move to that, I do ...

00:13:03.11 >> Yeah?

00:13:03.20 >> There's one other thing I think we're learning and which is also sort of ... informed the answer to this "What's next?" question is I think, as Kelley and I and our team just gets deeper and more familiar with how these collection of 17 indicators ... how they all inform one another ...

00:13:29.14 >> Mm-hmm?

00:13:29.42 >> ... how it really is, for all of sort of the headache and heartache sometimes these indicators cause, there is a wisdom and a brilliance to how they work together, and if we let them guide the work, things are going to get better for students with disabilities in our state. We are in full belief of that, and we're also learning as part of that coherence of these indicators that they can be used as a lever for other strategic priorities.

00:14:03.02 >> Mmm.

00:14:03.68 >> Specifically the work that we're leading across our agency with educational equity, I think letting these indicators and the requirements around them, letting them be a lever and not a hammer is one of the things, I think, that we're learning to do, and I'm really excited about what that's going to mean for sort of the energy and enthusiasm of the work going forward.

00:14:27.12 >> Yeah, that's really exciting and just hearing you ... It gets me happy listening to you saying that because it is so often we get so deep in the weeds of the indicators and not thinking about how they work together. How can we really be using these to our advantage and to really make a difference and not looking at it as just like, "Ugh, this something that we have to do every year," and you just do it for the purpose of doing it because it's required?

00:14:56.94 >> Right. We've even, in our more giddy moments, kind of thought about the SPP as our superpower rather than as our beast of burden, right?

00:15:06.47 >> Yeah, yes. Yes, absolutely.

00:15:09.26 >> As far as our plans to continue engaging stakeholders, again, like we said, we really want to focus on increasing our number in diversity of stakeholders, and we just continue to add to our internal and external stakeholder meetings, and those connections that we're making are becoming more broad-reaching. So just different advocacy groups that we may ... hadn't thought of before have been reaching out, and so that's exciting to us, and we're hoping to just see those meetings grow. And as we already said, we're going to be doing an indicator progress highlight at each of those meetings just to keep those indicators front and center and really for them to understand the progress and how it works together with the other work that we present at those meetings around our SIMR and around our educational equity work. So we'll try to weave that in so that it really does make sense with the work that we're doing. And also one of the things that we have been doing and will continue to do is that we will analyze all of our indicators by subgroups, even the ones where OSEP doesn't require us to break it down by race or by disability or by other subgroups, and we've taken advantage, probably too much advantage, but using tools like Tableau and providing dashboards to our districts, so that they can see their data broken down in a different way and they're not having to create their own charts and things like that of data that we already have. Lauren kind of had the light bulb moment as we were considering giving our districts their data for their State-Identified Measurable Result. "Can't we do this in a dashboard?" And once we did, it was life-changing, not only for us but for our PSUs, and so continuing to provide data in an easily accessible readable format and then making changes when they ask for it has been super helpful.

00:17:09.24 >> Yeah, I think we should have said at the beginning, I think one of our most important and cherished stakeholder groups is our local Directors of Special Education.

00:17:19.99 >> Mm-hmm.

00:17:20.35 >> We do, in addition to our sort of formal external stakeholder group for the SPP, we have other formats for engaging those directors as a stakeholder group, multiple formats, and I think we continue to ... We're continuing to evolve in understanding how to use our indicator data as an indicator of the level of a support a given LEA would need, right?

00:17:49.25 >> Mm-hmm.

00:17:49.49 >> Sort of like what DIBELS is to MTSS, for us the indicators are to the proximity and the intensity of our general supervision, and we still have a long ways to go in doing that in a more systematic and organized kind of tiered level of support, but we're on our way and being able to conceptualize the indicator data in that way, I think, is going to be very organizing. I think the ... Finally, the thing that I think we're learning and are going to spend time talking about and focusing on with stakeholders across all different kinds of our stakeholder groups is really thinking about collective efficacy and our belief systems, and I say that because I think until we have a broad and consistent culture of high expectations across all eds and specifically for students with disabilities, I think we're going to struggle to see these data move in a meaningful and lasting way, and that's a really different kind of work, implementation science or improvement science.

00:19:03.64 >> Mm-hmm.

00:19:04.50 >> We're in the midst of quite a bit of research around, what are the evidence-based practices for building collective efficacy and for addressing belief systems and ... Yeah, we're excited about where that might take us.

00:19:18.18 >> Yeah, it's a whole culture shift.

00:19:20.51 >> Yes.

00:19:20.89 >> Wow! Well, you all have a tremendous amount going on, and I'm so excited about all of it. This has been amazing. Thank you for sharing. I think a lot of other states out there will really learn a lot and be able to take and ... some of the examples and strategies and try applying them themselves, so thank you, both, for being on.

00:19:45.64 >> Aw, thanks so much for letting us blather.

00:19:48.44 >> Yeah, anytime, anytime. Thank you.

00:19:52.33 >> Yeah, I think that any state that's crazy enough to do the kinds of things that our state has done in the past couple of years, we're here to support that.

00:20:02.06 >> Yes.

00:20:02.23 >> Wonderful. Thank you, both, so much for being on and sharing with us your SPP/APR journey and how stakeholders have been a part of the process from the start and have continued and will be continuing into the future.

00:20:18.45 >> Thank you.

00:20:18.90 >> Thanks, Amy.

00:20:21.06 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###
00:00:01.51 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.43 >> Welcome to "A Date with Data." Today, I am joined by Kelley Blas, Part B Data Manager, and Lauren Houlihan, the SSIP Coordinator, who are both with the North Carolina Department of Public Instruction. We know that states are required to engage stakeholders throughout the entire SPP/APR process, and Kelley and Lauren are here to talk about how they've been ensuring stakeholders are meaningfully part of the process on an ongoing basis. Thank you, both, so much for being here, and I'd like to start out, if you could each say a little bit about yourselves and your role. Kelley, do you want to go first?

00:01:03.36 >> Sure. So again, I am considered right now the Part B Data Manager. I've also taken on the role as the SPP/APR Coordinator. I have been with the Department of Public Instruction for going on 17 years now, and about 15 of that was solely working as the Part B Data Manager, and we had a separate SPP/APR Coordinator, and she retired about 2 years ago, right when the measurement table changed, and I had a huge learning curve in studying these targets and baselines for the new measurement table, but it has really been an adventure for me, so happy to be here. Thank you for inviting us.

00:01:48.90 >> Yeah, thank you. Lauren, how about you?

00:01:51.71 >> Yeah. I have been with the Department of Public Instruction since 2007, so 16 years, about the same amount of time and similar journeys in terms of rural transitions here at the department. I started as the Consultant for Occupational Therapy, and then our SSIP Coordinator at the time, Matt Hoskins, became our Assistant Director. I moved into the SSIP Coordinator role about the same time that Kelley moved into the SPP/APR Coordinator role, and so we have definitely learned and kind of grown up together in our understanding and our coordination of our state's efforts around the SPP/APR.

00:02:37.19 >> Yeah, it must be nice to kind of have someone to go through that with in a similar way.

00:02:41.95 >> Well, and I'll say not just someone, but I have Kelley Blas.

00:02:45.79 >> Yes.

00:02:46.03 [ Chatter ]

00:02:46.03 >> Say no more.

00:02:47.18 >> She's just in it, and an incredible, incredible ... one of the more productive people I've ever met.

00:02:54.79 >> Oh, that's wonderful. Great. Thank you. So just to get started, can you talk about how you've been engaging stakeholders in the SPP/APR process?

00:03:09.33 >> Sure, I'll start out with that. Obviously, our journey began with the new measurement table, and so right away, we knew that we were going to need a lot of organization and support as we started this journey to reset the baselines and targets for our SPP/APR [Indistinct]. So we immediately contacted our IDC person, Rachel, and we started off by using the ... I think it was called the Planning Template, and maybe you can correct me if I'm wrong, but there was an outline for us that allowed us to kind of plan what we were going to do in that process, and part of that was setting our time line and how we were going to hold these meetings to involve our stakeholders in resetting our baseline brackets, and so that was a very helpful tool, very helpful meeting for us in deciding how we were going to do it, and I'll let Lauren kind of help with the outline of those meetings and how we brought people together.

00:04:19.82 >> Yeah, it seems like a really long time ago, but I guess it was just really 2020. I think part of ... That was also at the beginning. Our journey started sort of with COVID too, and so just thinking about how to do all this work in a virtual space. It was a different kind of coordination. So we really ... That summer, fall of 2020 and on through until we got really through 2021 because this was for the FY 2021 submission, right?

00:05:00.07 >> Mm-hmm.

00:05:00.48 >> We're 2 years into the new measurement table. And so we held all of our stakeholder meetings to develop the new SPP online. We did a pretty wide invitation, wide-open invitation. We worked with lots of our different advocacy organizations and our local contacts with our LEAs, just to make the information about when the meetings were, and we just tried to cast a really broad net.

00:05:37.86 >> Mm-hmm.

00:05:39.32 >> We had ASL interpretation and Spanish interpretation for all of those meetings. We did multiple kind of repeated sessions on individual indicators. We recorded all of those sessions, posted all of the recordings, kind of kept that information in front of our partner organizations and our LEAs.

00:06:02.33 >> Mm-hmm.

00:06:03.37 >> I think one of the things that ... Well, one is, we didn't know what we were doing, and we had no idea what it meant to just throw wide the invitation, right, to ... There were several of those meetings where we had 75 or 80 people on the call.

00:06:23.77 >> Wow.

00:06:23.91 >> I think one of the things that I've said in the past, some of the maybe confidence or maybe just foolishness to do that was ... Our leadership in the Office of Exceptional Children, several years ago, invested in facilitator training for most of the OEC consultants, so we've been through pretty rigorous facilitation training so that we can conduct focus groups and large meetings and move groups through decision-making processes, through analyses of problems or situations. We have structured protocols for that, and I think just being equipped in that way really set us up for some surprising success and maybe taking some risks in terms of that open invitation that we might not have been so comfortable trying that if we had not had that training.

00:07:31.95 >> Wow. That's really cool that you all have that training.

00:07:34.78 >> Yup.

00:07:35.02 >> It seems like, duh, everyone should have that, especially given, yeah, the stakeholder engagement work and how much you would need to be facilitating those meetings and making the most of people's times and learning those strategies to really engage and get productive conversations going. So that's really cool.

00:07:57.62 >> Yup, yup.

00:07:58.62 >> And what are some ways that you've, kind of during the meetings when you've brought the stakeholders in, that you've helped to build their capacity to really be able to provide that meaningful and useful feedback?

00:08:15.26 >> I'll start off on this one again and then toss it over to Lauren, but one of the things that we did right off the bat was, as soon as we got the measurement table, we began setting up data displays for each indicator, and so what that looked like was, we would take at least 5 years of historical data and look at that, and then we would show them what it would look like with the change. For example, in Tier 1, the whole dataset changed, so we showed them what it looked like from before and what it would look like after, and we did some backtracking with the after as well, just to try to give them some meat to what they were looking at in a full understanding of what that indicator was about and what the change was about, and we went really, really slow, so I don't think we had more than two indicators at time per month, so we went really slow. We talked through all of the changes, and we also provided them with some options for their baseline and their targets. We didn't just say it has to be one of these. We also offered for them to have some conversation and provide some feedback if there was an option that we didn't think of. So we really tried to engage them in thinking about what the meaning of the indicator was, how it impacted our schools and our districts and our state data and then also allowed them to think about how they would set their targets, and so that seemed to be very helpful, I think. There was, and we may get into this in challenges, but I'll just go ahead and say it. I think at times, we may have provided too much data.

00:10:03.43 >> Mm-hmm.

00:10:06.44 >> And there's times where we felt and got feedback that they were a little lost in what we provided, so when that would happen, we've been trying to go back and analyze it. Lauren, what do you think?

00:10:19.91 >> Yeah. I just think, and a couple of things to add to building stakeholder capacity: We definitely had this kind of whatever it was, 16-month schedule of how we were going to move with our stakeholders through these indicators. We had a plan. And I remember pretty distinctly, I think it was summer, July of '21, we got to Indicator 3, and just the particular group of folks that we had, they're like, "You're asking us to make decisions about stuff that we don't understand well enough, and we just got to slow down." There was just some ...

00:10:58.12 >> Mm-hmm.

00:10:58.61 >> And because ... What, there are 16 different baselines and targets with Indicator 3 now.

00:11:04.18 >> Yeah.

00:11:05.59 >> And we were like, okay. We just need to slow down. We're going to take another month. We're going to back up. We're going to explain what each of these different parts of Indicator 3 mean. We're going to give folks more information ahead of time, and we kind of ... We revised our schedule. We let people know we were responding to the request, and so I think just listening and being able to ... being willing to adjust empowered those participants to just kind of stay with us and to not be threatened by people's sort of emotion. I think that was another ... We just created space and made adjustments.

00:11:46.95 >> Mm-hmm.

00:11:47.78 >> The other thing I'll say in terms of stakeholder capacity is particularly for those stakeholders who just showed up month after month after month. WE extended invitations to lots of those folks to join our what are now SPP workgroups. We have four different workgroups: data literacy, stakeholder and family engagement, research and informed practices, which we ... and then our systems coherence, which we sort of modeled after the National Center of Systemic Improvement, one of their frameworks.

00:12:25.31 >> Mm-hmm.

00:12:25.66 >> Those workgroups aren't just DPI employees, and they now have external stakeholders that participate in them, and in fact, we have coleaders that are external stakeholders on a couple of those workgroups.

00:12:38.90 >> Hmm.

00:12:38.99 >> And so folks really, I think, continue to deepen their understanding of the Indicators and the work in terms of improvement science that goes along with actually seeing those data improve over time, so we're really excited about that level of ... And our capacity has been built because of that level of engagement on workgroups.

00:13:05.98 >> Yes, for sure.

00:13:07.54 >> Yeah.

00:13:08.41 >> That's such a perfect example of how really taking those stakeholders and bringing them in, not just at that target setting stage but carrying through to talking about the data and the results and improvement strategies and all those other elements, so it sounds like that was a really great strategy to bring in folks who had taken part in that target setting and bring them in to these ongoing continuous workgroups now.

00:13:38.72 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.51 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.72 >> Hello, and welcome to "A Date with Data." During this episode, we are continuing our incredibly interesting discussion with Dawn Kemp and Dominique Donaldson, who are both Part B Data Managers with the Georgia Department of Education. They shared on the last episode what their stakeholder engagement looked like over the last few years, how it's evolved, and now, we're going to hear more from them in terms of strategies that they have found to really help keep the stakeholders engaged and also what they have coming up next for stakeholder engagement in Georgia.

00:01:04.04 What are some ways, and how do you think you've been able to maintain this ongoing feedback from stakeholders, going beyond the target-setting piece and really, like you've said, getting more into the improvement activities, evaluating progress, analysis. How has that, you think, been able to be so successful in your state?

00:01:29.57 >> I think that Dominique can speak to that in terms of our intentional strategies that we've used.

00:01:37.37 >> So as far as maintaining those intentional strategies, we really ... In each stakeholder session, we really had a focus, so we would focus on specific indicators and making sure that we were continuously working towards presenting the SPP/APR information in a manner that was clear and understandable to stakeholders. Making sure that we created a safe space for the discussion allowed us to have those deep discussions where they felt comfortable questioning and asking in order to gain a very in-depth understanding of the content so that we could use that information to inform our decision-making. We also ... One of the other things that we did was scheduled stakeholder sessions in advance. That helped keep the momentum going. What we allowed people to do was register for the sessions, and when they registered, they were sent a confirmation email. And they were able to add the event to their calendar with reminders about the event. And so we found that if the stakeholders have the event on their calendar, it increased the chances that they would actually attend the session and then, as we also talked about, intentionally adding Spanish interpreters and soliciting participation through fliers in Spanish.

00:03:09.25 >> Great. That's a really smart idea, making sure you include that calendar invite, so it's not ... It's much easier to get it on their calendar rather than someone having to remember to put it on there, and they don't always do that. And what are some ways that you follow up on any concerns that maybe come up during the stakeholder engagement discussions by bringing in other people to kind of help ... some experts in those areas to really help answer some of the questions that maybe come up about the different initiatives?

00:03:43.36 >> So Dawn had previously mentioned how deep the discussions had gotten.

00:03:48.34 >> Mm-hmm.

00:03:48.60 >> And so because the discussion is so rich, sometimes we do have questions or topics that require us to dig deeper and continue the conversation, and what we would do then is when the stakeholders have those questions about students with disabilities, such as students with the most significant cognitive disabilities, we bring in program specialists to present in our SAP, and they provide and clarify information. Other topics and areas that also had a lot of interest included transition in preschool.

00:04:25.34 >> Mm-hmm.

00:04:25.60 >> And so we also brought in program specialists from our Division of Special Education Services and Supports, and they have also presented information.

00:04:35.51 >> I'm sure ... This is a very difficult undertaking and was for all states with the new package, and like Dawn mentioned at the beginning, that really explicit charge from OSEP that this be an ongoing, continuous, meaningful initiative bringing in stakeholders to provide input going beyond just the target setting. What are some of the bigger challenges that you have faced with this process of bringing in these stakeholders and getting that meaningful input?

00:05:09.91 >> One of the challenges we experienced was scheduling. It's difficult to find a time that works for everyone, but we did offer multiple sessions on several days with offerings in both the morning and the afternoon in order to get the best possible wide range of stakeholder participation. Another challenge we had was making sure that the information that we talk about and disseminate is done in an understandable manner that's bite-size, so people can really conceptualize what we're talking about to be able to use the information and really participate in the discussion.

00:05:53.11 One of the other things that was very important was, how do we keep the open communication with stakeholders when we're talking about setting targets? We all have high expectations for our students. However, communicating the difference between our expectations and the state targets can be somewhat challenging.

00:06:14.88 >> Mm-hmm.

00:06:15.45 >> When we set targets, we think about those targets of being attainable. So when we review data, targets should be made based on reasonable or attainable growths or progress. Sometimes, in the discussion, there are unrealistic expectations that can yield unrealistic targets. If targets are unrealistic and never attained, there's a loss of enthusiasm, and it could minimize the progress that's actually being made. So there's a hard line of when you have high but not realistic expectations.

00:06:52.92 >> Yeah, I can see that as definitely being a challenge when you're working with the stakeholders who, of course, want 100 percent, and where you're at is at 20 percent. And, yeah, so that could be a challenge, for sure, with that discrepancy between, really, the ideal and what, of course, we hope and want for our students but what, like you said, is realistic and attainable and achievable and making kind of those smaller, incremental steps to improve.

00:07:23.72 >> Right, and like what you mentioned, working through some of that process when we're looking at, for example, assessment results ...

00:07:32.65 >> Mm-hmm.

00:07:32.95 >> ... they may not be exactly where we want them to be. So when we're thinking about creating achievable, smart targets for assessment, that may ... When we're looking at data, the stakeholder may have emotion ... It may generate an emotional response when we're trying to really use data to inform the decision on where the target should be, and that can create some kind of angst sometimes, and we have to just work through it, be very intentional about working through the desire for our students and what we ... Looking at the data, where the data tells us we need to go based on rate of progress and ultimate goals for what can be reasonably attained within a year or within a certain amount of time.

00:08:32.19 >> Yeah, and that really leads into talking about those strategies and initiatives and programs and getting the stakeholders. Yes, we want a lot of progress and hope that we eventually get there, but let's talk about how we're going to get there and what ideas they have, too, for progress. Coming in now to 2023 and moving forward, what are some changes you have in mind? What are your plans for continuing this wonderful stakeholder engagement that you've been able to maintain now for several years?

00:09:13.10 >> So we really want to capitalize on our collaboration with Parent to Parent and increase our collaborative efforts. We found that collaborating with Parent to Parent, we're able to increase the amount of parent participation from groups of people that we normally would not have reached. Also, we want to provide additional survey opportunities. We discussed how in the meetings, people may not be ready to vocalize their thoughts. However, when given the opportunity to submit a survey, which is an anonymous survey, we can really collect honest feedback, and we can use that feedback to inform our decision-making. Also, we want to really look at increasing our marketing of stakeholder input sessions, so that's a huge priority for us. So we plan to create a marketing campaign that will disseminate information about our stakeholder input sessions through various avenues. And our plan, so far, includes providing information in invitations through our Special Education Directors' Webinar, weekly newsletters, collaborative partnerships such as Parent to Parent, our website and professional learning activities such as our budget and data professional learning, and that's just to name a few.

00:10:38.44 >> And Parent to Parent, that's your state's parent information center?

00:10:44.50 >> Yes.

00:10:44.86 >> Yes, ma'am. Go ahead, Dawn.

00:10:47.87 >> Yes, I was going to say yes, yes, it is. And they are ... They have representation on our State Advisory Panel and have shown a tremendous willingness to assist in this work.

00:11:04.61 >> Great. It's always so nice to see in states where there is that strong collaboration between the SEA and the parent center. You can see, especially with stakeholder engagement, how much of a difference, really, that can make. Is there anything else either of you want to add that you didn't already discuss? Any other highlights or tips, strategies you want to share with us?

00:11:29.76 >> I just ... One thing that comes to mind is that when you have a really diverse group of people, like our State Advisory Panel is, you have to be very flexible when you're presenting.

00:11:43.85 >> Mm-hmm.

00:11:44.06 >> Just like what Dominique said about providing information in bite-sized or in manageable chunks, you have to be able to do that. You also have to be flexible enough to work with those participants on the State Advisory Panel that have not statistical knowledge, understand psychometrics, ask really tough questions, questions some of the formulas that are used. So it's a ... It really requires a lot of flexibility so that you're not shutting down people that are not as interested in that level of topic and that you are, in fact, providing a good, relevant response to those people who are asking the technical questions and/or offering feedback for that. Sometimes, when you have those technical type individuals, it's helpful to provide them with follow-up, say, in terms of graphics or charts that they can look at, following the meeting or at a different time, so that you're not stopped completely on maybe a technical point.

00:13:09.58 >> Yeah, that's a great point. It's really a balancing act when you have a group that kind of ranges the spectrum of folks who maybe are pretty new to all of this and ones who have that more technical background, like you said. And that's such a really great idea to kind of answer questions but then say, "We can definitely provide you with more information, more detail afterwards," or even have follow-up conversations with those maybe who want that deeper discussion.

00:13:41.26 >> And I would also add that start early. Start your planning for stakeholder feedback sessions and engagement now.

00:13:51.67 >> Yeah.

00:13:52.88 >> And then also, let your stakeholders know that their feedback was used to make the decisions that were made.

00:14:03.22 >> Mm-hmm.

00:14:03.58 >> So communicate that, "We considered this. This is what we made a decision based on your feedback."

00:14:12.13 >> Yes, 100 percent. Yeah. When you do have ... The next time you meet with them, leading up to it, make sure you're telling them how useful and critical what they've provided has been, so that they hopefully will continue.

00:14:24.21 Well, thank you both so much. You shared such a huge amount of information. I'm still kind of reeling trying to unpack it all in my mind, but this was really informative. Thank you both.

00:14:37.16 >> Thank you for the opportunity.

00:14:38.08 >> Thank you.

00:14:39.86 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date With Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.56 >> Thank you for listening to this episode of "A Date With Data." I am so happy to welcome Dawn Kemp and Dominique Donaldson, who are both Part B Data Managers with the Georgia Department of Education. And they are going to share with us some of the work they have been doing to continuously and meaningfully engage stakeholders in setting targets, but also going beyond that and analyzing data, developing improvement strategies and even more. And I want to just to start out by asking you each to say a little bit about yourselves and your role and your background. Dawn, do you want to start off?

00:01:02.78 >> Certainly, Amy. My name is Dawn Kemp, and as you've already stated, I'm one of the Part B Data Managers for Georgia. I have over 35 years of experience in public education. I have worked in a variety of capacities. I've worked as a classroom teacher. I've worked as a special ed lead in a school building. I have worked as a college and university instructor. I have worked with regular curriculum teachers in alternative preparation programs. Additionally, have worked as a special ed director in both Georgia and Alabama, and most recently, across the last 3 years, I have served as Part B Data Manager in the state of Georgia.

00:02:00.56 >> Thank you. And I didn't even know all that, and I've known you for a few years now, so that was good to hear. How about you, Dominique?

00:02:08.91 >> Hi, my name is Dominique Donaldson, and I am a Part B Data Manager for GaDOE. I work with Dawn Kemp, and I have had about 10 years of experience in education. Through my time, I've served as a special education teacher. I also served on the district level as a lead LEA facilitator as well as a school-based LEA facilitator and also as a program manager. And on the collegiate level, I also served as a university instructor.

00:02:44.65 >> Wow, sounds like you and Dawn actually have similar backgrounds. Well, thank you both. The first thing I wanted to ask about is, can you just tell us what your stakeholder engagement looked like in 2022 and kind of how did that all get started? I know you kind of were doing a lot of work in 2021 and kind of continuing and building off of that into 2022. So what has that looked like over the last couple of years?

00:03:11.46 >> Thank you so much, Amy, for the question. Examining the stakeholder engagement in '21 and '22, it has been a continuous process. And what I mean by that is, obviously, the stakeholder engagement in '21 was directed a lot by the guidance in the SPP/APR measurement template and the guidance from OSEP about the genuine and increased stakeholder engagement and feedback that was required.

00:03:51.36 >> Mm-hmm.

00:03:51.84 >> It obviously, from reading all the documents provided by OSEP, was highly clear that this was not a cursory level of engagement or checking a box. That you were genuinely, as a state agency, compelled to get meaningful feedback from your stakeholders. And across the last 2 years, that has looked somewhat different. In the first year, it was really like a tsunami of effort because prior to that, there had been stakeholder engagement predominantly limited to the State Advisory Panel, which is a very diverse and well-created panel. But it's not like getting feedback across the state from multiple regions, from ... and especially more from parents. So if you look at the charge that LE ... that state agencies were given with the SPP/APR 2020-2025 package, you realize that you had to really have some pretty strong scaffolds in place to make your stakeholders engaged partners. And I began that work in 2021 by taking the SPP/APR and really clustering or chunking the indicators into specific subsets of indicators, so as not to overwhelm participants. And I divided it into clusters such as group A was where your indicators were high school and life outcomes, and that include your Indicator 1, which was your graduation rate, your two, which was your dropout, 13, which was your transition type plans and 14, which was your post-school outcomes. So that was one group of indicators that had a commonality and that I felt could be explored by a group of stakeholders in a meaningful way, such that they would be able to actually dig into the data, understand what the data meant across multiple years, examine those trends and assist the state agency, that would be the Georgia Department of Education, in setting meaningful targets. So group A was just one of the clusters. There was also a group B, which was disproportionality, and that included Indicators 4, 9 and 10. Now, the majority of those indicators are compliance indicators, so you're not looking at targets, per se. But when you begin to discuss a topic like disproportionality with a group of stakeholders, they want to see the entire package, and that is why it included the entirety of those indicators. Group C examined environments for educational services for children that were school age and children that were younger, the preschool ...

00:07:48.50 >> Mm-hmm.

00:07:49.19 >> ... age. And it also looked at your timelines indicator. Again, a preponderance of ... or a number of those are actually compliance indicators that are set by OSEP, particularly for 11 and 12. But it ... You need the stakeholder engagement to determine methodologies or ways that you can make improvement. Then, group D examined preschool and parent involvement and legal outcomes. And then, group E was the assessment. It was only Indicator 3, and group E, the assessment, has become, by far, the most challenging in terms of attaining stakeholder feedback in ... Not engagement. The stakeholders have been highly engaged, but answering some fairly technical questions and looking at, as we'll discuss later, the target setting for Indicator 3. So when we began the process with those clusters of indicators, we took our State Advisory Panel, and we asked them in 2021 which of these indicator sets, which of these clusters do you have a personal interest in? In other words, would you like to do more intense and intentional work?

00:09:29.34 >> Mm-hmm.

00:09:29.85 >> And we actually allowed the stakeholders on the State Advisory Panel ... At that time, it had 37 parent members.

00:09:39.22 >> Wow.

00:09:39.28 >> And of course, there were additional members beside the parent members.

00:09:43.82 >> That's a lot.

00:09:44.26 >> And we allowed them to basically sign up to work with a particular cluster. We brought in specialists from the Department of Special Ed Services to facilitate each cluster.

00:10:03.38 >> Okay.

00:10:03.78 >> And they were able to share the data with these different cluster groups, assist in setting draft targets and examine practices that were taking place within Georgia. I think it is important to note that to make the work uniform, each cluster group had a presentation and a format of work that was designed and developed ... I developed it so that each group had the same layout in terms of understanding what baselines are, understanding what targets are, looking at the data within their specific area and then examining ways that they could set targets. So that was highly uniform for each of those groups.

00:11:07.93 >> Okay.

00:11:08.15 >> And so we met with that work in the year 2021, actually September of 2021. Now, we also expanded that work beyond the State Advisory Panel to include multiple other sessions that were conducted virtually that we solicited feedback from parent groups, from teacher groups, from people that were in regional technical assistance, state department level personnel. And we conducted additional webinars on these ... all of the indicators. So 2021 was an extremely busy year, and there were ...

00:11:59.86 >> How many of those did you do?

00:12:01.82 >> I think I ended up doing in excess of 20 meetings ...

00:12:06.99 >> Wow.

00:12:08.60 >> ... across all of the different groups. And we also generated feedback and engagement through the use of a survey that we sent to everybody that participated, to give us feedback on the targets. We hypothesized that some people might not be comfortable talking in a group.

00:12:34.20 >> Mm-hmm.

00:12:34.64 >> But they might be willing to give you a survey, an anonymous survey result.

00:12:41.38 >> Yeah, and also they might have needed time to kind of think it over and consider the discussion and then send you some feedback after. So that makes a lot of sense.

00:12:49.84 >> Absolutely. But we also kept ... We wanted to be sure that we were looking at representativeness.

00:12:56.53 >> Yeah.

00:12:56.63 >> So each of the surveys, we did ask the participant in the survey not to give a name or anything or to identify an LEA or anything of that nature.

00:13:10.38 >> Mm-hmm.

00:13:10.63 >> We did ask them for their race or ethnicity.

00:13:14.19 >> Mm-hmm.

00:13:14.58 >> And we did ask them for what their role was. Were they a parent? Were they a teacher? Were they a state level administrator? And we also allowed for them to capture more than one role because some people are parents of students with disabilities, but they also may work as a teacher in an LEA. They also ... So those types of things.

00:13:36.94 >> And how representative ... Did you get a pretty good mix of participants, would you say?

00:13:44.06 >> We did. Dominique, I believe you were going to speak a bit about the representativeness?

00:13:52.05 >> Right. It was pretty representative of our stakeholders, but we found that there was a little bit of underrepresentation in the Hispanic group.

00:14:03.01 >> Hmm.

00:14:04.54 >> And because of that, we did add additional stakeholder sessions with Spanish-speaking translators, and we also developed fliers to reach the unrepresented group to help bring those represented stakeholders into the conversation and give them more access.

00:14:25.45 >> Great. Was that successful? Were you able to increase ...

00:14:29.36 >> Well, we hope that ...

00:14:30.37 >> ... the families?

00:14:30.74 >> ... it's a little bit more successful in the upcoming year ...

00:14:33.00 >> Okay.

00:14:33.29 >> ... with more collaboration with Parent to Parent, other groups like that, to get the word out more. We didn't necessarily yield the result, but we definitely laid the foundation for increasing the stakeholder support for the upcoming year.

00:14:49.60 >> Awesome.

00:14:50.42 >> Dawn, how ... You kind of laid the foundation, I think, for 2021. How has it now evolved? What kind of happened last year in 2022?

00:15:00.80 >> We continued with ... Obviously, we laid the groundwork, so when we began in 2022, instead of having to go back through protracted discussions, we were able to just kind of pick up where we left off. And we did, in September of 2022, we reviewed that SPP/APR framework. We provided all available data on the indicators that we had. Obviously, some of the indicators, such as Indicator 3, were not available at that time. But if we had data, we began to provide that to the stakeholders, and we continued in our discussions with them about the data. And we also continued in our discussions with the stakeholders beyond the State Advisory Panel, and we conducted additional webinars, once again, to try to get additional feedback. In addition, we continued to have survey that was available for all participants. We tried to encourage, as Dominique mentioned, more of a collaboration with Parent to Parent and also with our parent mentor group at the Georgia Department of Education. One thing that I would say that has happened with our stakeholder engagement and our engagement that's markedly different in '22 than '21, is that the discussion with the State Advisory Panel has become extremely deep. There's a lot of depth in the conversation. They have an understanding of the indicators to such a degree that they ask questions about formulas for how you derived, particularly, the assessment indicators, discussions of that nature. And they also want to know more about instructional initiatives that are attached to making improvements in areas such as the assessment results of students with disabilities.

00:17:37.49 >> Thank you both so much, Dawn and Dominique. It was great to have you on and just hear so much about your story of stakeholder engagement, how it's evolved and how you're going to be making improvements and what it's going to look like in the future.

00:17:55.38 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date With Data," with your host, Amy Bitterman.

00:00:07.34 >> Hey, it's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff, who, just like you, are dealing with IDEA Data every day.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.52 >> Thank you for joining this episode of "A Date With Data." I am chatting with Audrey Rudick, who is an IDC TA provider and we'll be discussing the topic of EDFacts modernization. You may have heard about EDFacts modernization and maybe are wondering what it is, or you've heard something about it and really aren't quite sure how the changes might impact you and the work and the data. Well, Audrey is here to help demystify all of that for us. So welcome, Audrey.

00:00:56.70 >> Thanks, Amy. It's great to be here.

00:01:00.05 >> Can you start out just saying a little bit about yourself and your background?

00:01:04.24 >> Sure, well, like you said, my name is Audrey Rudick, and I'm a TA provider with IDC, and my background is really with partner support. So I was on Partner Support Center for a number of years before joining IDC, oh, just over 3 years ago now.

00:01:21.31 >> Great, so you are well-immersed in the world of EDFacts and 618 Data, for sure.

00:01:26.96 >> Yes, yes, I'm pretty familiar with 618 Data and with EDFacts and the ins and outs of how those data are going through the system into ED.

00:01:38.24 >> Wonderful. Well, let's start out by just hearing from you what is EDFacts modernization? We've been hearing those terms and maybe some of us don't know what that is.

00:01:51.03 >> Well, absolutely. So the US Department of Ed collects data through a system called ESS, and ESS is old enough, maybe to vote at this point. It's pretty old in terms of data systems.

00:02:06.73 >> Mm-hmm.

00:02:07.12 >> And so EDFacts modernization is really their initiative to bring EDFacts ... the EDFacts data collection up to date. EDPass is going to help update that data collection in a few ways. One way is that it will allow states to manage their own users and offer multifactor identification, which will make it more secure.

00:02:31.09 >> Hmm.

00:02:31.99 >> And it's also going to allow states to manage their own data quality process. So it's going to modernize that data quality process so that it's really giving ownership to data quality to the state and ensure that the data are ready for use and publication by the due date.

00:02:47.73 >> Great, yeah, I think that's one of the most exciting parts about it, is when the data gets submitted, not too long thereafter or kind of immediately, then it's ready for use and aren't waiting months at a time before that data can be put to use for all the reasons that we know about that we're collecting it for.

00:03:06.60 >> Oh, absolutely. I think right now you submit files in November for some of those files, and then you find out that they are ... the data quality results in January, and then you have until May to update them or add data notes, and it really draws everything out. I think that putting it all into one process is going to make life, in the long run, much easier for everybody.

00:03:32.45 >> Yes, we just got to kind of get through this transition, which I know some states are kind of anxious about, because it is definitely a change in processes and timing. So can you talk about how the modernization is going to impact states as well as the IDEA data that are being submitted?

00:03:53.23 >> Absolutely, so for more of the IDEA data, there's no changes to the actual data that are being submitted.

00:04:01.31 >> Mm-hmm.

00:04:02.14 >> In in the most recent package, they've clarified some of the definitions. But really that wouldn't change anything that's being submitted. The biggest impacts are going to come through timelines because the deadlines are changing. So for example, we're recording this in February of 2023, and typically states would be working on submitting their child-count files pretty soon because they would be due the first Wednesday in April. But for child-count files this year, they're going to be due starting June 28th, after a 5-1/2-week-long submission window.

00:04:49.71 >> So it really pushes it out, at least for this first year.

00:04:54.82 >> Yeah, I think that they will be on those annual dates after this. So the timelines are changing for those files that are typically due in fall and winter, that's going to look like assessment files being due January 10th of 2024 instead of in mid-December. And then exiting discipline and personnel files being due February 22nd of 2024. So it's pushing it out quite a while for those files. So I think that because of those due date changes, some processes are going to change. The other thing that's changing with those due dates, and we're creating that bigger space to submit those files, is because the data quality review is happening at the time of the data submission. So in the new system, states will upload their data. They'll update load those files. Then checks will be run almost automatically to return data quality concerns against the business rules that have been published for those files. And with those checks that come back, then before submitting those data on the deadline, you'll have to enter data quality notes or you'll have to reupload the data to correct those data quality concerns. So that 5-1/2-week ... There's going to be a 5-1/2-week timeline for that data submission, and that 5-1/2-week timeline will become important to kind of make sure we're not waiting until the end of that in case any updates are necessary.

00:06:44.99 >> Yeah, having ... putting it in as soon as it opens, or as soon as possible, really, seems like it would be beneficial because inevitably, I would imagine a lot of the files there will be issues that arise and maybe are fine and you can put in your data quality note. But you might have to go back to districts to kind of clarify and correct things, and that takes more time.

00:07:11.40 >> Absolutely, absolutely, and I think right now, we know quite a bit about how this process will work, but we're all going to be learning it together. There's a lot that we don't quite know yet since we just haven't seen it. But I think that reviewing your processes ahead of time and saying, "Okay, we want this person and want our data manager to review all of those data quality concerns that are returned within this new tool and create those data notes. And this is going to be our process for going back to the LEAs to make sure that we know what we're putting in those data notes is correct." Those process reviews are going to be really important so there's no surprises about who might be doing what during that time period.

00:08:03.52 >> And when, so really, yeah, thinking through ...

00:08:05.28 >> Yes.

00:08:05.59 >> ... them now and having that kind of ready to go.

00:08:10.57 >> Yeah, yeah, the other big implication that I can think of really, is working ... our audience, I suspect is a lot of Part B data managers, but creating that work with your EDFacts coordinator because the EDFacts coordinator is listed as the first EDPass manager. And that's a new term for us, EDPass manager. The EDPass manager is the person in the state who is going to own the access points for EDPass. So you'll be working with that person to make sure that you can enter those data notes or to review the data or to upload the data, and making sure that they know what you need during this time is important because I don't think that every program office that has data collected through EDFacts is going to have the same ownership of data that we have found that special education offices have.

00:09:22.34 >> And all data managers will be getting access to EDPass, right, but it has to come through the EDFacts coordinator.

00:09:30.10 >> Not automatically, all data managers are going to need to talk with their EDPass coordinators to advocate for that access and to ask for that access ...

00:09:40.88 >> Okay.

00:09:41.21 >> ... because those EDPass managers are going to determine who needs access and who will receive access for the state.

00:09:50.54 >> Hmm. Are there other recommendations you have for how states can be preparing for EDPass and these new processes and timelines?

00:10:00.18 >> Well, I think that reviewing your data processes, which we've discussed, and making sure that we're aware, determining how you're going to store those data quality notes as they come in, as you see, "Oh, this LEA's data is going to look a little bit differently this year." And then having those conversations between your EDFacts coordinators.

00:10:26.19 >> Mm-hmm.

00:10:26.40 >> I guess one thing I forgot to mention is the metadata piece. So instead of completing the state supplemental survey, which typically happened in February, states will be working on those responses alongside of the data collection in EDPass. So EDPass will ... if you have those mismatches between your metadata and your data, in EDPass you'll be able to collect that as you're submitting and reviewing the data.

00:11:00.22 >> So that's one big advantage ...

00:11:03.17 >> Mm-hmm.

00:11:04.01 >> ... in just being able to enter that data, too, at the same time. So your questions related to child count would be due at the same time as your child-count files.

00:11:13.58 >> Exactly, and the discipline questions ...

00:11:15.72 >> Mm-hmm.

00:11:17.45 >> ... and exiting questions will be due at the same time as the discipline and exiting. The other thing that I think that you could do now, if you've agreed that you're going to have access, multifactor identification for EDPass is going to be through login.gov. And to ensure that that is set up by the time your EDPass manager might be granting you access, creating that login.gov account with your e-mail address that you use for your state work will be important so that you have that multifactor identification ready to go.

00:11:55.46 >> So anyone can create an account on login.gov? Or do we need some kind of permission or invite?

00:12:03.16 >> Anybody can, that's multifactor identification for, I think, all of the federal government.

00:12:10.20 >> Mm-hmm.

00:12:10.57 >> But that will be used for ... specifically for EDPass and our purposes in this modernization work.

00:12:20.50 >> And we also, I don't know if, Audrey, did you want to mention about the upcoming EDFacts data manager peer group?

00:12:28.60 >> Sure, thanks, Amy. So the IDC is also going to be hosting a Data Manager and EDFacts Coordinator peer-to-peer exchange. So if your role in a state is IDEA Part B data manager or EDFacts coordinator, you're welcome to join our exchange. Our first convening is going to be on March 9th, and we're going to be really just discussing how we're working together, what might need to happen in the state and what everybody's needs are in terms of the data and data submissions.

00:13:09.67 >> So I think with states ... anyone listening need to reach out to their state liaisons, or is there a way on the IDC website?

00:13:16.07 >> You can reach out to your state liaison or you can contact me directly. My e-mail address is audrey.rudick@aemcorp.com.

00:13:25.72 >> Wonderful. Hopefully, this kind of helped solve some of the mystery around EDFacts modernization. But there's going to be much, much more to come as the system opens up in May for the IDEA, first IDEA files, child count. And I know IDC will be definitely putting out more resources and providing technical assistance throughout all of this and be talking about it on numerous calls. So this is not the last you have heard. There's just going to be a continued ongoing conversation. So we're really excited. And thank you, Audrey, as our expert in all things EDFacts and EDPass. So thank you so much for sharing your wisdom and thoughts with us.

00:14:10.93 >> Absolutely, thank you so much for having me today, Amy.

00:14:14.96 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Resources:DMS Resources for Grantees - Individuals with Disabilities Education Act DMS 2.0 Document Review & Request Template SEA Data Processes Toolkit | IDC - IDEA Data Center
TA Center Differentiated Monitoring and Support (DMS) 2.0 Adapted Protocols

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.66 >> Welcome to "A Date with Data." Today, we are changing it up a bit, and I am so excited to be joined by two of my colleagues. We have two IDC TA Providers: Mary Watson and Amber Store. And they have been really involved with several of the states who are in Cohort 1 of OSEP's Differentiated Monitoring and Support, or DMS, and they are joining us to talk about the experience that they've seen and lessons that they've learned from their perspective as TA providers, and to get started, could each of you tell just a little bit about yourselves and your background? And, Mary, do you want to go first?

00:01:04.63 >> Sure, Amy. Thank you. My name is Mary Watson, and I started working with the IDEA Data Center, IDC, in its first year of being launched, and prior to that time, I had worked with the North Carolina Department of Public Instruction in various roles in special education. In my last years there, I was in the State Director of Special Education.

00:01:28.96 >> Thanks, Mary. Amber, how about you?

00:01:31.61 >> Sure. I'm very happy to be here with you all today, and my experience is a little bit opposite of Mary's, where Mary was here at the very beginning of IDC, I've been with IDC for just over 1 year. But prior to that, I was a research and data specialist at the West Virginia Department of Education for almost 10 years, so I come with a very strong background in evaluation and data and analytics.

00:02:04.91 >> Thanks. Yeah, I forgot, both of you have that strong history in the states before you became TA providers, so I know that probably helped out a lot in the work you've been doing with states, especially with DMS.

00:02:17.13 >> Absolutely.

00:02:19.38 >> So this first question, I'm just going to ask Mary because she, I know, has been involved with some of the states who are in Cohort 1, really, since Phase 1. Can you tell us how you have been supporting the states that you've worked with during Phase 1? And just as a reminder for folks who may not really be familiar with DMS, that's the phase in which states are providing relevant information back to OSEP, documentation responding to questions and that sort of thing. So, Mary, what was that first phase like for you in working with the states?

00:02:54.90 >> Well, Amy, when we first started working the states in Phase 1, and I advised this even moving forward, it's important, I think, to build your team as TA providers, and we did that across centers with IDC, partnering within CSI and Cipher to get to know each other to support the state and to get to know the context of the state. So you're kind of building your TA team, but you're also getting to know the state team that's going to be working with the DMS because depending on the size of the state, every single staff person may not be directly involved, so you spend a little bit of time getting to know who's going to be involved and how the state is going to approach their DMS visit. And I think from there, many conversations about: This is what the state needs to be doing every day. The DMS visit is about checking the state's systems, but it's not above and beyond what the state should already have in place. And I think that's important for all of us to understand. In saying that, in the actual technical assistance, we looked at their systems, an overview of their systems. And using some of the OSEP provided resources like the Document Review and Request Template and the OSEP Protocols, we started to kind of help the state organize themselves with the people that were going to be involved around those protocols and then reviewing their website or helping them to review their website to make sure that they have the documents that are being requested and/or also to make sure that things on the website are current and accurate, that they are not out of date, and in many instances, we haven't done that many yet, but as states start reviewing their websites, they do find things that need to be taken down from the website.

00:05:02.86 >> Hmm.

00:05:04.14 >> We supported the state during the OSEP calls and interviews, and during that phase, we're mainly listeners and notetakers, but immediately following the calls, we would follow right back up with the state for a debrief to help them get ready for the next call or help them strategize how they're going to respond to some of the questions or things they were asked in the call. So really, just being kind of partners with the state, silent partners, for the most part, while they were working with OSEP but then helping them to prepare their materials, helping them to be ready for the next interview, maybe asking some questions to help them prepare for responding during those interviews that were taking place during Phase 1 that may be a little different moving forward. And encouraging the state to feel comfortable and let OSEP know when they need to have some time to get back to them on certain questions. If they're in the Phase 1 interviews, and they really aren't sure about how they should respond, they need to just say that, and then we'll help them draft a response, or they can look through their documents and make sure they have what OSEP is asking for.

00:06:22.72 >> Yeah. That's really great advice. I think in the midst of it when you're kind of nervous or anxious, and you want to just respond and give them an answer, you just have to keep remembering to give yourself that time to take a breath and think, and you can always say, "Let me have a chance to pull that together and get it back to you." So then kind of moving in to Phases 2 and 3, which is now called the engagement phase, and also during the visits themselves, Amber, can you start off by talking to us about the support that you provided in those Phases 2 and 3?

00:07:00.30 >> Yeah, absolutely. So having come on board after Phase 1, Phase 2, and then of course, Phase 3 is when I really kicked in, in working with my state. So one of the first things that the state did, we had a really strong state team, and they had already submitted a lot of their documents, all of their documents to OSEP, and so one of the next things that we did was update all of their Data Process Toolkits. The Data Process Toolkits are IDC tools that allows states to capture their processes for the different data collections, knowing the time lines to the people responsible to the file specs and locations, and this is really ... It was really an internal activity for the state. It wasn't something that they did to share with OSEP. It was more an exercise to make sure that all of their protocols and websites and data displays were all up to date so that they did have everything in place as OSEP was going through and checking their documentation and checking their website links. And so the state just found that as a really, really helpful exercise to get them prepared.

00:08:24.04 >> Mm-hmm.

00:08:25.26 >> Then a couple of other things that I was more heavily involved with is, whenever we would have a meeting with OSEP, we would always come together with different TA providers and the state team and have a debriefing call, and it was usually very, very brief, but it was a really good way for everybody to kind of synthesize what was going on, what the next steps were so that nothing was missed. I found that incredibly helpful because as not part of the state itself, I was able to observe and perceive things in a different way than when you're the state being put on the spot at that moment.

00:09:09.19 >> Mm-hmm.

00:09:09.85 >> And so it was just a really great time for everybody to be able to reflect, and then the other big piece was a site visit that we did in preparation for their OSEP site visit, and this kind of goes back to what Mary was saying in that we were really lucky in that the state invited TA providers from not only IDC, which was myself, but also from two other technical assistance centers, the NCSI and Cipher, and so we had three different TA providers from three different organizations with three very different perspectives that were able to all help coach and suggest and keep thoughts in the forefront and provide some techniques that really helped the state become a lot more comfortable and confident, and they had all of their pieces in place, but, again, just the practice and these different perspectives that we brought to the table, it really put them in a position to succeed. They were a lot more comfortable and prepared to talk to OSEP during the site visit.

00:10:24.62 >> Mary, what about you?

00:10:26.79 >> Well, I certainly agree with everything that Amber shared about Phase 2. I think being that friendly face that they're used to seeing during Phrase 2, especially their own site, was more helpful than I realized. They articulated that, that having a friend in the room was helpful to help them relax. Listening, again, and taking notes so that you can, at breaks, or when you debrief, as Amber talked about, the debrief times being so valuable, you can help them remember some of the things they may need to mention. The states may hear the question from OSEP and not really answer directly, as many things that they're doing that they need to share, I don't know if I said that clearly, but I think the TA providers, in listening and taking notes, can help the states articulate. Remember, you are using this strategy, and you need to tell them about that, so you can do that coaching that Amber was referring to. And also during Phase 2, when they were on-site, I started, along with others that were there, to just jot notes about how we would support the state following the visit as you hear the questions and you know what materials they have and maybe where they need more support. It's a good time to just start drafting out some things that you can offer in support in a follow-up manner.

00:11:52.20 >> All the TA providers that have been involved have been supporting states. I know I've heard from a number of states who are maybe in Cohorts 2 or later for the DMS visits that are completing those data process protocols that you mentioned, or updating them if they've already done them. You mentioned already a couple of the tools and resources, I know, that are available. Mary, what have you found to be the more helpful for states in preparing and taking part in the on-site or virtual visits?

00:12:24.21 >> Well, definitely states that have completed or are in process of documenting their data processes has been extremely helpful, and Amber spoke to that as well.

00:12:34.35 >> Mm-hmm.

00:12:35.00 >> It's helpful to have that documentation, but the conversation that that process elicits is so important in helping states prepare to be able to articulate their system to anybody else. There's just lots of rich conversations as you document all of the 618 Data and the 616 Data, so it has the great outcome of having your data processes documented, but it also has the unseen outcome of helping states talk about their system and understand it better, and then along with that, the OSEP-provided materials, the document review and request template, I found, as a TA provider, to be very helpful. It gave me a starting place on, "Oh, these are the documents they're going to want to see," so when you meet with the state, you have kind of a list of what documents they're going to be looking for. So I thought that was helpful. And then OSEP has protocols for each of the areas like compliance, the data area, the fiscal area. They have protocols with questions that they may ask. If it's not guaranteed, they'll ask 100 percent of those questions, but those are extremely helpful, and then the TA centers took those protocols, working together with the TA centers, and made them into a checklist format so that it's the same questions, same material, but on the right column, there are areas where you can check off: If you have this additional information, the state may want to record, so they're kind of working documents. I found those to be very, very helpful.

00:14:13.89 >> Yeah, and we'll put links to all of these different resources too in the notes for the episode.

00:14:18.86 >> That's great.

00:14:20.61 >> Amber, any other resources or tools that you want to mention?

00:14:25.50 >> Beyond the things that Mary has mentioned, I would just reiterate that we used the protocols from OSEP as well in our preparation, and we really leveraged those to make sure that everybody kind of knew who was going on the be taking the lead in speaking, and then we would have someone who would be kind of the secondary speaker, and the state actually, for those different sections within the protocols, they actually kind of took the time to write out almost script, and they didn't read that verbatim, but they did read it out loud to each other so that they could make sure that they were providing the best possible response to each of the questions, to each of the prompts, and I think that really ... Not only did it help with the comfort and confidence and actually articulating their responses to OSEP, but they became more practiced at it, and the team together collectively became more knowledgeable about each other's specific roles within their larger system, so I think that that allowed everybody just to have a much better idea about the big picture as well as understanding of their individual responsibilities and roles.

00:15:51.21 >> And finally, what would you say are some of kind of the key lessons learned, advice or tips that you would share with states who are maybe in Cohort 2 or Cohort 1 and are still in kind of those earlier phases? Amber, do you want to kick that off?

00:16:09.23 >> Yeah, absolutely. One of the things that we've informally been working on, and it might maybe, knock on wood, someday turn in to a formal tool, is kind of putting together a check sheet, a cheat sheet, a quick facts or things to keep in mind type of document, and allowing the team to set forth their own protocols for the things that they were going to do in the meeting. Again, it's kind of like going back to, who's going to be the primary speaker, and who's going to be the secondary speaker? And then when possible, what we found was an extremely helpful tool that goes along with that is having some sort of live chat, whether that be through a computer program or on people's mobile or cell phones, having a group text going during calls and meetings, as long as it's appropriately done in a way that's not going to be distracting or offensive, but that allowed all the team members to say, "Don't forget to say this. Don't forget to say that," and that's where the secondary speaker really came in, because the primary speaker, you can't expect them to keep up with the chat. They're the ones that are being put on the spot in that moment, and you can't always expect that person to remember 100 percent of everything that they want to mention.

00:17:36.78 >> Mm-hmm.

00:17:37.17 >> And so that's why the secondary speaker would be able to follow along with the chat and be able to insert themselves in an appropriate place and say, "Oh, so and so, don't you remember that this, X, Y, Z, is part of the system?" or whatever it is that they're trying to explain and really be able to kind of make sure that there weren't gaps.

00:17:59.57 >> Mm-hmm.

00:18:00.34 >> And so that was really, really helpful. We found that practice does make perfect. We did mock interviews, and that seemed to help a lot. We stressed that it's best to try to be as succinct as possible and sort of only answer what is asked, and that is not, in any sense, a way to not share as much information as possible. It is that you're really ... The state is really, really trying to paint that picture of their systems for OSEP, and as we all know, they're very complicated systems, and you have so many interchanging working pieces that we can equate to puzzle pieces or cogs. There's just so many moving parts that it can be kind of hard to paint that picture for outsiders, and so that was one of the things that we kind of stressed is: Answer exactly what is being asked. And if you have a good example, or a lot of times, you want to be able to show evidence or proof.

00:19:08.88 >> Mm-hmm.

00:19:09.28 >> And so being able to point to where that exists online, if it is a publicly available document. Things like that are awesome, so please add those things in. But go too far beyond what they're asking because you don't want to muddy the waters. You want to be able to help them truly develop that overarching picture of what the state system is like and where the evidence exists.

00:19:34.94 >> Something I've heard too is, it's not enough just to make a statement or say, "Yes, we do this," or ...

00:19:40.16 >> Mm-hmm.

00:19:41.42 >> Even explaining kind of how, you really have to have that documentation that's either publicly available or internal or the evidence to show, "We have these letters that we send out for notifications. Here's what they look like."

00:19:55.18 >> Yeah. And if you were to see my head right now, I'm nodding emphatically that a lot of it is being able to show, exactly what you're saying, to show the evidence, and finally, I would say ... And I think most people are really excellent about this. I know that the states that I've worked with are ... You have to be patient. Monitoring is not a simple process, and a lot of times, even if a state were to look through the OSEP comps ...

00:20:26.81 >> Mm-hmm.

00:20:27.15 >> ... you can see that a lot of the questions are kind of repetitive, but they're also almost tiered, where you have the overarching question, and then it goes into ... The next question goes into a little bit more detail. The question after that goes into a little bit more detail beyond that, and while it feels that it's repetitive, again, I just think that everybody has to keep in mind that you're trying to provide an overview and then also some very specific information about complicated systems, and just to be patient and to kind of expect some of that repetition but to just always remember that it is in an effort to see the big picture. It is in the effort to be able to say, "Yes. The state has this process and this evidence in place," so they're just trying to get just the big picture, and so, yeah, my advice would be to be patient.

00:21:29.37 >> Yeah. You might have to answer what seems to be the same question multiple times or in slightly different ways, but it's these ... OSEP isn't familiar with all the details, and they're just trying to understand and learn as much as they can. Great. Mary, what tips, lessons learned, key advice do you think is good for states to know?

00:21:53.29 >> Well, I would definitely reiterate what Amber just ended with: answer what you're asked and no more. Don't elaborate because in elaborating, you may get off topic, but just stay with the question and be as positive as you can be and do not hesitate to ask OSEP to restate the question ...

00:22:12.78 >> Mm-hmm.

00:22:13.31 >> ... to make sure that you heard it clearly. You don't have to do that every time, but if you need additional time to think, or if you want them to restate the question, it's fine to ask, and you can just tell them if you need more time, or you want to get back to them at another time on that topic. That's absolutely fine. When you know the date that OSEP is coming, I would advise ... A tip is to talk to your stakeholders that are going to be involved and your state advisory panel, your EC directors, other groups that you believe OSEP may be calling to let them know that you are going through this differentiated monitoring and support process, that you're excited about it as a state, or you're preparing for it as a state, and you want them to be as aware as they can be. I would hope those partnerships are ongoing, but in any case, I would notify your stakeholders and be as positive about the message as you can be. And I think it's helpful if you think about ... If OSEP is coming on-site, if you think about logistics ahead of time, where are you going to position them in the room, and where are you going to have your staff in the room? Who's going to be talking, and does everybody need to be there all the time? Or do you need people going and coming for some of the parts? I just think, thinking logistically about how it's going to work, where people can hear the best, looking at the room you're going to be in, and is OSEP going to have people that are calling and part of the virtual conversation? So that brings in a technology aspect. Does your room have Internet access? Are you going to be able to have the Zoom application or Teams application so that OSEP can participate virtually? If that is true, and even if it's not live, they pretty much always have a virtual component or somebody who's still in DC who didn't come on-site. But it's helpful to have a technology person right there because if you have to stop because somebody can't get on the Internet, or you can't ... The Zoom isn't working, it's really handy if your technology person can be on-site. That's just another tip that popped into my head. As far as the interviews, also, when OSEP is on-site, and you have designated person, and as Amber said, maybe a follow-up person to answer questions, if you ... As a state, you should not have contradictory ... It's not a good time for you to argue about something, so don't override or contradict what somebody else has said. That's what your TA facilitators are there for. We can go over that after the fact in those debrief sessions we talked about, but you should not contradict each other during the interviews, and you think that would be just the norm, but if somebody hears something they disagree with, sometimes they just want to say something right in that moment, and it's not typically the best time. And back to the logistics part, is there an area where people can get coffee or water? And just those creature comforts that you may want to think about when you have guests coming to visit your state. You want your state to look good. You want to receive OSEP in a comfortable atmosphere and perhaps give them suggestions. Sometimes, they tell you what they want to do for lunch, but if not, you may want to provide a list of restaurants that are close by that can help facilitate their needs. Those are the main ones that I can add to what Amber said. She gave some great ones also.

00:25:48.56 >> Yeah. Now, I'll piggyback off of that a little bit because I think it's kind of funny that the logistics of in-person meetings are something that we kind of have to bring forefront again, but after several years of mostly having virtual meetings, I think those are some really great points, Mary. And I would only add to that, just be flexible because the state that I worked with, the on-site visit had to be from ... It had to transition from a face-to-face to a completely virtual, and so the state had to pivot because they were already really kind of set up for the face-to-face, and it all still went really, really well, but just being flexible to whatever the meeting situation ends up being. That's just kind of the world that we live in today, and remember that you might plan for one way, and then it might go another way, but just to keep rolling with it.

00:26:49.18 >> Those are some great tips, and kind of some takeaways I heard is, like you said, the flexibility and just all the preparation, hours and hours of TA providers' time and of course the state, just to really get ready and to make sure that they're very thoughtful and have everything laid out and discussed and just that planning and just gave us, I think, so many great ideas and things to consider. Thank you, both, so much for sharing your wisdom, and love to have you on again maybe when you've gone through a few more of these and have even more that you've learned as things change. We know this process is evolving too. So thank you, both, so much.

00:27:35.05 >> Thank you, Amy.

00:27:35.99 >> Yup, my pleasure. Again, all I can say is practice makes perfect, and it pays off, and it pays off in dividends. It really does because of the pressure for the situation. But like playing a sport or doing anything else, the more practice you have, the better you are at it, and the better your results are going to be. So thank you for the opportunity for us to speak with you today.

00:28:01.16 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.85 >> Welcome to "A Date with Data." Today, I am joined by Matt Sewell, Director of Special Programs; Yvonne Greene, Administrator of Monitoring and Program Effectiveness; and Jody Fields, the Part B Data Manager, who are all with the Arkansas Department of Education. Arkansas is one of the first states who have now gone through Phases I and II of OSEP's differentiated monitoring and support, or DMS, 2.0 system. Just as a refresher, Phase I is when states are providing relevant information in response to OSEP's request for documentation, and then in Phase II is when the states are interviewed, and there's also an on-site or virtual visit, and we're going to hear all about Arkansas' DMS experience so far.

00:01:14.01 So to kick things off, could each of you just say a little bit about yourselves? Matt, do you mind starting us off?

00:01:20.99 >> I would be glad to. My name is Matt Sewell. As she said, I'm the State Director of Special Programs. Prior to coming to the Department of Education, I spent almost 20 years in the public school system in a variety of roles, ranging from classroom teacher, school-bus driver, coach, director of finance, athletic director, special education director, building principal prior to coming to the department to fill the role of Special Education Director, and now, I'm in my fifth year here at the department.

00:01:58.53 >> Wow. You've really done everything.

00:02:00.82 >> I've done a little bit of a lot of stuff, and I've never been really good at any of it, in fact.

00:02:08.00 >> I'm sure that's not true. Oh, my gosh. Okay. Jody, how about you?

00:02:14.54 >> Hi. I'm Jody Fields, and I have been the Part B Data Manager for Arkansas since December of 2003. So that is 19 years and 20 child counts.

00:02:29.25 >> But who's counting, right?

00:02:30.45 >> Who's counting, right? So I am actually housed at the University of Arkansas at Little Rock with a grant through the Special Education Office to handle the special education data management, and my office basically handles all of it in regards to the APR and the 618 Data.

00:02:50.95 >> Great. Thank you. Yvonne?

00:02:53.52 >> Hi. I'm Yvonne Greene, and I am the Administrator for Monitoring and Program Effectiveness. I've been with the Department here for about 10 years in a variety of roles, and prior to coming to the state of Arkansas, I was an educator in Texas. My experience is in special education and general education in a variety of settings, resource, self-contained and inclusive settings. And I was also an early childhood coordinator.

00:03:24.49 >> Great. Thank you, all. So to get things going, we'll start kind of at the beginning, and if you could just tell us how you got ready for DMS, what you did to prepare for that first phase of sort of documentation and pulling together all that information that OSEP requested? And then also moving into that second phase of your interviews and your virtual visit.

00:03:49.68 >> Great. I don't mind to kick us off there. So we started this process actually in the fall of 2019, when this was first launched, and at that time, what we did and what I would most likely recommend others do is, first and foremost, look at what it is that OSEP is going to want. Those, the protocols are available for everyone to look at, and after you get a hold of those protocols, really think about and strategically think about, who are the people that need to be involved? Is it ... Do you have an administrator of monitoring? Do you have people who wear multiple hats? Who are the people that you would like to speak on behalf of your agency? Those are all pieces of that thought process when you begin, trying to determine how you're going to navigate everything that there is to collect because there's a lot of stuff there. Once you identify your team, you need to start pulling your team together and meeting with them, making sure that whoever is going to be responsible for each part has exactly what they need to move forward and start collecting that data, and I would, at this time, I'd really like to invite Yvonne and Jody to speak in to some of this part of the process too because they really did the in-the-weeds piece. You've got kind of the overview part of thinking through who your team is, what do you need? But then there's a lot of in-the-weeds work that you're going to need your team member to dig in to.

00:05:30.74 >> Yeah.

00:05:30.83 >> I found that the protocols that were adapted by the TA centers were especially helpful because they added columns for you to talk about who was responsible for particular sets of information, where documentation is to support whatever the question is asking, and there was a way on those adapted tools to really answer whatever the protocols questions were. I found it very helpful to go really deep, understanding that there was a lot of repetition in some of the potential subquestions because they're really trying to seek clarification and understanding of what your procedures and processes are. And so if you've written solid answers, sometimes, your solid answer may apply to multiple questions because, again, they're trying to go deep. But at the same time, you have to hold the overarching question from the protocol in mind so that you don't go down extra rabbit holes. So what is it that they are looking for in this particular question? If they're looking at the integrated monitoring questions that keep the conversation around how your team truly integrates, they're monitoring protocols and procedures across all of the areas within their office.

00:06:59.71 >> So one of the things I think we all did was look at our actual internal procedure manuals, finance, monitoring, data, dispute resolution on how we ... what we had in place, what could've been missing, whether there's things that we haven't updated. So when it came to data, we had done the Data Process Toolkit from IDC 5 years ago, and in the spring of '21, we started to update all of those. So going through and updating them and making sure we had things in place that we knew what our procedures were, especially since there was new staff. You hadn't been here for only a year at the time as we were getting into this, once it kicked back off the second time around.

00:07:48.09 >> Mm-hmm.

00:07:48.81 >> But really is making sure you know where everything is, and one of the other things we went digging into was our website and making sure the website was up to date, that we had the most recent information out there. So as OSEP went out looking for things, we could tell them exactly where it was and that it was the most current information.

00:08:11.90 >> And I think that really helped with the documentation request because we were able to have so much available on our website that it limited what extra information we needed to go find because we have so much available publicly.

00:08:28.85 >> And, Amy, I would add that as you develop your team, your team of individuals, one of the first topics of conversation, once you get past, "What documents do we need to identify? Where they are on our website, what about our process guides?" is, what TA centers need to be part of this team? And who are those individuals from those TA centers that you should immediately involve? Not wait until you've begun the process, but allow them to be part of your team from the beginning, from the onset of the process until the completion of the process? Because the questions that they'll ask you, the things that they'll take a look at for you, just being your teammates and coaching you at times through it was invaluable to us.

00:09:18.24 >> Yeah, I'm sure as more states are going through this, also having TA centers who have been through it with others states, will be helpful because they'll know the experience, know what types of questions might be asked.

00:09:29.11 >> One of the other things, I would say, is even if they're meeting on finance, have everybody there because they're going to bring up finance. And so for data, they'll bring up the MOE CEIS reporting, so your data manager may need to be there, or we're looking at: If someone was noncompliant in something, monitoring still needs to be there. So while you're thinking it's just this particular topic, it all is integrated, and it all overlaps. And so you think, "I don't need to have somebody on this call," and then in the end, you're going, "We should've had so and so on the call with us."

00:10:05.72 >> Mm-hmm.

00:10:06.63 >> Because it does overlap across, and that's what they're really looking for, to see how we are integrated and that we all work together.

00:10:16.12 >> Yup, and that everyone, it's not just one person and very siloed, but it's a team effort, and it all comes together, all the pieces, like you mentioned. Yvonne, were you going to say something?

00:10:28.17 >> Yeah. We used the TA centers to review our protocol guides or any procedure manuals that we have just to help them help us. They were reading it as somebody with a fresh set of eyes and were able to provide feedback, which helped us clarify and actually have a stronger procedure manual process guide in whatever particular area that is. So they really ... The TA centers really helped us identify some holes that we were able to fill in prior to jumping into Phase II.

00:11:02.48 >> It sounds like you took those protocols, and did you actually kind of, like Matt was talking about, assign different people to their protocol that fit in their work? And then kind of went through and actually drafted responses or kind of jotted, put down, "Here's the evidence we have to support each of these items"? Was that part of the process?

00:11:22.84 >> Yes.

00:11:22.94 >> It was absolutely part of the process. Each person went through their protocol individually, and then we would meet collectively to go through those protocols to discuss what examples you had, where those things were located, and just to re-emphasize what Jody said, when you meet, meet together. There's some work that can be done individually, but then come back together for those discussions because as Yvonne shared about whether it was her process guide, where things were located, items that she found needed to be updated, it sparked either other questions or comments from other team members that these were connected. Here's how we can connect them, or, "Oh, yeah, we need to do that for this area as well." And it allowed for also some questions that you may also get from OSEP or from your TA centers. It's opportunities for Yvonne to ask Jody questions or me to ask Yvonne questions that just reinforces that collaboration and organization of your system, which is extremely important to OSEP, and Jody said this, how is all of it integrated? Because you can't work in these silos within your state. You've got to be more connected for the process to work.

00:12:45.04 >> Yeah. As you're talking about it, it's interesting because it's sort of almost like an unintended consequence of going through this, similar to kind of the IDEA Data processes where, part of it is being able to come together and collaborate and hear what each other are doing and how you work together and being able to ask each other those questions. It's sort of ... Really, even just going through it strengthens that relationship and your collaboration. Are there other resources? You mentioned the protocols and the TA, modified templates are something things you used when you were preparing. Are there other resources you use that you found helpful?

00:13:24.10 >> This isn't a written resource, but building off of what Matt said about utilizing the TA centers. We had a team from the TA centers we use most frequently come in and sit with us for a day and a half or 2 days and actually coach us through what the experience of OSEP being on-site might be like, and again, all of our team member who could be available and needed for the interviews were available for part of that process, and that really helped us, again, hear different ways of saying things or building that cohesion between our different groups to make sure that we really are all on the same page and have that integrated piece.

00:14:11.71 >> For the business ...

00:14:12.81 >> And there were some things that we needed to update from 1920 into '22, '23, and we had a new ... One of our team member was brand-new, so just being able to go back and review where we were then, where we are now was very beneficial. And again, don't ... When you think about resources, you have your protocols. You've got your TA centers, but your team, your staff, there's a lot of resource right there. Looking at some historical pieces, Dr. Fields has been with us for, as she said, 20 child counts, and that's how from, moving forward, how I'm going to refer to how long she's been here, by child counts, not year.

00:14:57.67 >> Yeah.

00:14:58.65 >> But just looking at some of those historical pieces too, where you've been and connecting them to where you are is a very beneficial resource to have also.

00:15:08.75 >> Yeah, because I'm sure if there's questions that OSEP has like, "Why did you change this?" or, "They are going to go back potentially up to 3 years," I think, right, in the questions and document? And having that context and background and perspective is so critical.

00:15:25.40 >> Yes. Yes.

00:15:26.49 >> I think the other resource you have to remember is the rest of the department.

00:15:31.57 >> Oh, I'm glad you said that.

00:15:32.86 >> Mm-hmm. Yeah, say more about that.

00:15:34.13 >> Because, well, so for Arkansas, we have a single student management system, and all the districts use it, and so some of the documentation I was providing is the documentation for the entire student management system, not just special education, because it's not a standalone system and such. And the same with when were talking finance and audits, it's about, how do you handle this? Well, that comes from over there, or our allocations come from the department finance office, not special education's finance office. So sometimes, your team member needs to be on standby throughout this process.

00:16:13.49 >> Yeah.

00:16:13.86 >> Or other folks outside of the agencies or in another section of the agencies, and by outside the agencies, I mean our mediation center, which is funded by a grant over at the Bowen School of Law that we had to have calls with them, and we had them on standby if we needed them during the visit.

00:16:34.78 >> Yeah, I would say, thinking through each area, what tentacles does it have? Because as Jody said, with finance, we needed to pull in the greater ADE finance and their procedures. We needed to bring in legislative audit because our school districts in our state are required yearly to have some type of audit, and legislative audit, sometimes, does audit findings, and if those around the area of special education, what is your notification process? And then what kind of actions do you take? So really thinking through the resources that are outside of the Office of Special Education and the other folks that you work with just outside of the doors in the offices that you sit in every day, and, Jody, thank you for bringing that up, because like she's saying, Rick would dispute. Needing to connect with Bowen School of Law, who handles all of our mediation and facilitation processes. It's very valuable.

00:17:32.37 >> Yeah. That's a great tip for states that are still getting ready for their visit that they have to definitely think beyond just special education and, like, you said, those tentacles and where they reach and having those early conversations with them to make sure they're onboard and understand what this looks like and what they need to contribute. In terms of that, so you all had a virtual visit, I know. Can you talk about that experience? And what were the kinds of questions that OSEP was asking? Just kind of share with us what that was like.

00:18:03.80 >> So I would say that the virtual visit was very similar, and my team, they may have a totally different take on this, but I would say that the virtual visit was very similar to the calls that we had scheduled in Phase I. We had many 2-hour calls scheduled in Phase I where we were going through protocols or going through documents that had been submitted or looking at things from the website, and the virtual visit was very similar, except for, it was all day.

00:18:39.55 >> Mm-hmm.

00:18:40.25 >> Which is ... An all-day Zoom ... A 2-hour Zoom is tough. An all-day Zoom is a little tougher, but they were really gracious with it. They scheduled breaks in there, so we could take 15, 20-minute breaks, schedule a lunchtime. Honestly, I probably would've preferred to have people on-site because when you are looking at documents and trying to explain documents, sometimes it's easier to just place them in someone's hand so that they can be reviewed, and maybe sometimes, there's less questions or questioning because they do physically have that document in their hand that they can review.

00:19:22.15 >> I think the virtual experience, I agree, I would've preferred a face-to-face. It just allows for a different level of interaction, but the virtual experience ... I wasn't sure what to expect, but it still ended up being a positive experience. There are multiple people on the team for OSEP, and those team member are assigned specific roles and specific protocols. So the way the question and conversation flows depends on the personality of the person asking the questions, but regardless of who's asking the questions, one thing I would say is to be cognizant of your language and vocabulary so that you're sure we're all talking about the same thing and the definitions we're using are common definitions. Ask for clarification. Pause when you need to pause and ask for a repeated question so that you can really gather your thoughts. Sometimes, there's just an intensity in the questions that we experienced, but be prepared for that and just take a deep breath and move forward. Don't let an intense moment derail what you know you need to say to OSEP. Just ask for that clarification and collect your thoughts and then answer the question. And rely on your team member. One of us doesn't always know every answer. We would ping-pong with our answers when somebody else said, "Hey. This is also applicable to that question."

00:21:04.73 >> Yeah. And for the data piece of this, really kind of ... We had such a good conversation with our meeting prior, and one of the things they requested was to see the student management system. And so I just was able to share the screen and walk them through our training side of the student management system and explain how we collect everything, and it really showed them how the data is all integrated into the bigger system. And by doing it virtually while they have a copy of the documents, and they're asking questions about the documents, one thing we could do, though, is share it back on the screen and show them exactly what we were talking about. And one of the things that you definitely have to remember is, if you're submitting a document for a specific question they're asking the document for, make sure you know exactly what document you sent them. Because I know there was a few times Yvonne was like, "No. I sent you this document to answer this question, not to answer that question."

00:22:10.48 >> Ah.

00:22:11.12 >> Because they're getting confused as to what was being sent.

00:22:14.16 >> Mm-hmm.

00:22:14.53 >> So internally, you've got to make sure you know exactly why they ... Why you sent them the document and which question it was being asked for.

00:22:25.29 >> Right.

00:22:26.68 >> Because you're not there together to be able to say, "No. That was this one," and hand them a difficult document. Instead, it's really understanding how everything is and what you sent and know exactly why you sent it.

00:22:45.46 >> And I would re-emphasize what Yvonne was saying, and she didn't say it this way, but don't get frustrated. They're asking questions, and some of the questions are very good questions, and some questions do seem somewhat repetitive, and some questions are about one form that goes with ... or a document you submitted that may go with something else as to why you submitted it, but don't get frustrated. It's like she said: Answer the question. If it's not ... If that document is not the correct document for the question they're asking, clarify what is. And also, remember that they're asking you questions because they don't work in your state. So what you know and what you know you submitted, don't take the attitude that, "Well, it's right there. I submitted it to you." Take the attitude of: They don the really understand what we're doing in Arkansas because they don't work in Arkansas every day. And so being, I guess, respectful, so to say, and treating them like they're your colleagues and not treating them like they're out to get you may help as you progress through the monitoring cycle.

00:23:56.77 >> Mm-hmm.

00:23:58.19 >> Yeah, just like you're treating them as new staff.

00:24:01.02 >> Yeah, yeah.

00:24:01.36 >> You're just teaching and explaining, yeah, how everything operates.

00:24:06.45 >> And for me, the protocols were really helpful in filling out the answers and getting the understanding of how repetitive it was because then, I anticipated more repetitiveness during the actual Phase II, and writing down the answers in the protocols also helped me. It was like my teacher lesson plan. Like, "Okay. I've written it. I know what's going to come next."

00:24:31.26 >> Yeah.

00:24:31.40 >> And so that gave me just a more solid internalization, I guess, of what I felt like I needed to say. So that was just another benefit of using the protocols because it did come back in Phase II to help guide through the answers and the clarity they were seeking there.

00:24:51.58 >> So were they kind of generally working their way through the protocols during the in-person?

00:24:57.75 [ Chatter ]

00:24:58.71 >> Yes.

00:24:59.04 >> They were.

00:24:59.84 >> Yeah.

00:25:00.45 >> Yes.

00:25:01.26 >> Yeah, they're working their way through the protocols with some follow-up questions, just really trying to gain, really, very specific clarity on what you do as a state, given a certain scenario or this subset of questions or specific document.

00:25:20.88 >> And the one thing, Amy, for Arkansas, we were not asked to complete the protocols and submit them.

00:25:26.76 >> Mm-hmm.

00:25:28.17 >> So we were completing the protocols for ourselves, not necessarily to submit to OSEP.

00:25:35.48 >> But it sounds like, no matter what, if they're going to ask for them or not, still having them all completed and being able to respond is key.

00:25:44.28 >> Right. Correct.

00:25:44.75 >> You're going to be miles ahead if you will work as a team to complete those protocols and have evidence to support what your answers may be for those questions and determine, who's going to speak about those questions? When it's time to meet with OSEP.

00:26:02.66 >> Right. And some things will overlap, so there's not an actual protocol out there for significant disproportionality, but we were sent questions around significant disproportionality, and while my office handles the identification side, Yvonne's office gets to handle all the compliance side of that, and so as we got into discussions on significant disproportionality, it took both of us to go through it.

00:26:29.00 >> And it took our financial manager, Josh, as well.

00:26:31.51 >> Right.

00:26:31.76 >> Because he handles and lets districts know about where they are fiscally related to the expenditures on CCEIS.

00:26:38.96 >> Right. And that's part of the reason why some of the questions seem to be so redundant. You get from protocol to protocol because they're trying to get it from, here's the monitoring perspective of that question. Here's the finance perspective. Here's the data perspective, and here's the general perspective that they all overlap, and so the same question just seems like it's always coming back, but it's the different components within your general supervision and how it ties together.

00:27:10.75 >> Yeah. That's a really good point, and I think Yvonne made that earlier on. It looks like you're answering the same question, maybe over and over, but you have to think about it from the lens and the perspective of that particular component or overarching question that it's part of. So always kind of keeping that in the back of your mind. Any other tips or recommendations you have after having gone through this that you think would be good for other states to hear as they are in the earlier stages?

00:27:39.33 >> Yeah. My number one tip is: Don't make it up. Either have it, or you don't. You're either doing it, or you're not. Don't make it up. Don't say what we're going to do.

00:27:50.64 >> Mm-hmm.

00:27:51.22 >> Really be very direct on what it is you do, how it is you do it, what kind of collaborative systems you have it in place and how it all informs your integrated monitoring system for your state. You may share with them later what your plans are because as you've gone through the process, you might notice a gap or a hole that you need to fill or a better connection you need to make, and you may create the most beautiful plan to do that, and you may want to share that with them, but don't say you do it if you don't.

00:28:24.32 >> And then have documentation to support whatever it is you say you do.

00:28:29.75 >> And my last thing is, be ready to brag on yourself.

00:28:34.48 >> Yeah.

00:28:35.02 >> They asked us questions that were not related to the monitoring. They even told us ahead, nothing. We had a whole conversation about equity across the state and different state initiatives around that, and they said this really had nothing to do with the monitoring. It's just they wanted information. So be ready to get to brag on yourself, so what you're doing well and what's happening in your state that they may not know about.

00:29:02.62 >> Yeah, highlight the good stuff. Highlight the great work that your teams are doing. Take that opportunity if you have it.

00:29:12.86 >> All right. Anyone have any last thoughts or comments?

00:29:18.12 >> I don't think so. We just thank you for giving us an opportunity to share what happened in our state, what our experiences were, and we just hope that it's able to help someone else prepare for their monitoring, so thank you, Amy, for allowing us to be part of what you're producing to help other states.

00:29:37.55 >> Yeah. Thank you, all, so much. I was just thinking, while you were talking, I've heard a lot ... I haven't been personally involved yet with any states in DMS, but hearing you talk about it, I have the clearest picture I think I've had. It's always been a little fuzzy. It's one of those things, I think, until you've actually done it, you don't really know what it's like, but now I do feel like I have a really good understanding. So thank you for explaining it so clearly and articulately. I think states will appreciate that. Well, thank you, all, again, for your time, and this has been really great.

00:30:10.06 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:25.06 >> Hello, and welcome to "A Date with Data." Today, I am joined by Mary Lane, who is an Education Consultant with the New Hampshire Department of Education's Bureau of Special Education Support, and Michelle Lewis, Executive Director of the New Hampshire Parent Information Center. Mary and Michelle are here to talk about the great work they've been doing over the last few years to improve the quality of their Indicator 8 data, and that really started by stepping back and considering the why behind this indicator. Thank you, both, so much for being here. To get us started, could you each just share a little bit about yourselves. Mary, why don't you go ahead and go first?

00:01:06.00 >> Sure. So I've been working in the New Hampshire Department of Education for over 20 years, and along with Michelle, we've been working quite closely engaging families and schools together in building better partnerships.

00:01:21.79 >> Great. Thanks, Mary. Michelle, how about you?

00:01:24.67 >> So I've been working at the Parent Information Center for over 20 years, and I'm very committed, very passionate about family, school and community partnerships, and that really drives the work that I do. I have two children: one who's 16 and one who's 18. And my 18-year-old has a disability, and that really propelled me into the work, and, again, as Mary said, we've been working together for a number of years, and our most recent work, we're happy to share today.

00:01:57.59 >> Great. So let's just start out at the beginning, and why did you decide to do this work around Indicator 8? What was the impetus?

00:02:10.01 >> Sure. I'll start. So as an education consultant, back in 2004, many of us in every state were really struggling, wrapping out heads around Indicator 8, the Family Survey. And the reason is, we were learning some indicators were performance or results indicators, and some were compliance, and it felt as though this indicator was something very special. It was something that probably needed to be addressed across all indicators, and that said, from the very beginning, we knew we needed strong technical assistance. We knew that what we had been doing, where we based our survey around the monitoring process was not enough, and so what I was immediately felt was being helped by Michelle, who I'm going to ask her to kind of talk about it, was the strong feeling that the Parent Information Center in New Hampshire has about their responsibility about this work.

00:03:45.39 >> Yeah. I feel like we've had a long history of working together around Indicator 8, as Mary talked about, and I would say the Department of Ed really has engaged all stakeholders from the beginning in this work, and Mary highlighted some pieces about that. We really wanted to be responsive. My view of the state was that they wanted to be responsive to families, to school districts and to community organizations. The work right now that was started last year and continues, really, it was a perfect storm in terms of the why for us. The entire state is doing work around family engagement, in particular, with preschool development grant work, looking at early childhood governance and really leadership at the state level, including the governor, including the Department of Education is very invested in family ... I should say across all departments at the state level, who is very invested in family engagement, and so again, it was this perfect storm, the department's dedication from the beginning to really look at this indicator.

00:05:01.16 >> Yes, Michelle. I agree. One of the things, though, I'm learning on this journey is that trust and respect from families is so important, and that being it's because many times, bureaus or departments of education really haven't been at the best game of really putting themselves out about engaging families, and in particular, it really has felt many times as though we get to a certain level, and then it backs down, even when you have the leadership. And one of the things that has ... I see a lot of patience from the local level and from the organizations that support families is understanding how hard it is for us to really make sure we maintain a high standard of this work.

00:05:59.63 >> Yeah, and I had been fortunate enough to be involved with you all in moving this work forward, and that's been so exciting and can definitely attest to the fact that really, you are bringing parents to the table in really amazing ways. So can you share a little bit more about what that work has been like with the parent, with your stakeholder group and how, Mary, you from the state and other from the state have been working together with Michelle and the Parent Center to move it forward?

00:06:28.98 >> Absolutely. So from the beginning, our commissioner, Commissioner Frank Edelblut, has been disappointed in our overall response rate, and that said, he actually reached out to the Department of Education to ask for help, to ask for really technical assistance, and that's where Amy was able to help along with Joanna Bivens, with regards to telling us, "It's okay to start where we're at, and it's okay for me as an education consultant, who's been doing this for many years, to step back and listen and actually be on equal level as a stakeholder." That took a lot of me trusting that I could actually do this and that the districts and the families would actually see that as a way of earning respect.

00:07:29.44 >> Yeah, absolutely.

00:07:31.49 >> I think that was a really important piece, bringing in IDC to work with us at the core team level. When I'm saying core team, I mean the Department of Ed, myself and then the IDC folks to help us create a process to really dig in to this work. It was really key for us.

00:07:52.99 >> Exactly, and I will actually add to that by saying: Without the help of technical assistance, we wouldn't be in the place we're at. And what I mean by that is, we kept saying we need buy-in from the local level, from both the families and the schools. But until we actually had them all come together with equal voice, which meant a lot of organizing every meeting, setting goals, addressing why we are here, and getting everybody to be able to take the time to absorb it and really actually maybe being frustrated in some of it, but actually coming to the realization that if we're not all in this together, it doesn't work.

00:08:41.09 >> And I think for us, in designing this process, we really looked at representation from across the state, both of families and with school districts in terms of looking at our geographic representation and our cultural representation and really try to design something that would be representative and to kind of move us forward.

00:09:07.42 >> I agree, Michelle, and one of the things that I saw happening was the fact that people actually were able to voice what they felt that they probably weren't ever going to feel strong enough to do, such as, some parents said, I understand from a school point of view that some of this is hard, and it might be confusing, but we as parent can handle this. We just want to make sure that we are able and have the permission to do it. And then from that, after several meetings, you actually saw folks working together and really deliberating together and coming up with better decisions of what we believed was the definition of how families were engaged and schools were facilitating in the special education process. It's so interesting because every day, I look at something, and I went back to looking at the state performance plan and the indicators and looking at the tool kit, and in the definition of Indicator A, at one point, it listed all the various levels by which we should be thinking about how families are engaged: written prior notice, the eligibility, making sure that all the laws ... But in the end of the day, do we ever get to really talk about how we partner together, and do we ever really feel as though that's really helping us with addressing why we're doing it, how we're doing it, how folks were feeling and whether the results were things that we were really feeling was actually happening?

00:10:49.54 >> Yeah, well, hopefully resonates with others listening and thinking about your Indicator 8 and bringing in stakeholders to be part of that process that I, again, was fortunate enough to be part of with you all in New Hampshire and kind of having that series of meetings, bringing in the parent and other stakeholders to really talk through, why do we collect this data? What are we going to do with it? What does it mean to have schools facilitate parent engagement? Really starting from the beginning with all of that.

00:11:20.82 >> I think, Amy, from my perspective, another key piece that IDC worked with Mary and I initially on and then the whole group is really designing a process. So I think you had us really think through, what were we trying to accomplish? And really, we were being very responsive to the family, the district and the state's needs Mary mentioned. The commissioner was disappointed in our response rates and really wanted to look at that, and families expressed survey fatigue. Our survey for Indicator 8 was quite lengthy, and so we also realized that local administrators, knowledge of Indicator 8 wasn't perhaps what it needed to be, so we saw that as something we needed to work on, and IDC helped us design a four-part meeting series with a focused goal, which was to shorten the survey, which was really to define our why, really to look at the definition of schools' facilitation of parent engagement or parent involvement and really use that indicator tool kit in our work, and that was, to me, really key because it gave us a framework in which to work. It gave us a framework in which to engage stakeholders with a very specific purpose. It gave us a framework in which we knew what our outcome was, but we didn't necessarily know how we would get there. And so it allowed us to really engage from the beginning as a group and really build some relationships over that period of time, really figure out, what is it that we were trying to achieve?

00:13:21.71 >> I totally agree, and one of the things I'm learning right now ... Because Indicator 8 is part of a larger survey called the 603, which is our area code, New Hampshire Family Survey, and it actually includes ... It's about families and engaging families, but it includes school level, community level. It's a lengthy survey. What it really did, our work, did was open the door and pave the way for others in the department to engage the public around the purpose of the survey, engaging them on their opinion about the survey, giving them a voice. Our next round of looking at the larger survey is taking the model of what we did around Indicator 8 and moving it forward with the larger community. And it's interesting for me to be able to see it happening in reality, so it actually has made marked improvement about the way we are going to, in the future, work around this 603 survey. It also is bringing people not only just to know more about Indicator 8 but to ask the really critical questions around, how do I support it? I need work. I need training in this. We need tools. We need the ability to really model that we are working together, and instead of just a school saying, "Here's a survey. Take it," it's around, "This is what we worked at, and this is what we believe in," and I think it actually gives it much more realistic and optimistic opportunities for families and schools to really come together and help improve outcomes for the children, which is what it's all about.

00:15:22.51 >> Yeah. That's so exciting to hear that you're going to be using this model and expanding it within the department. Are there certain areas that you thought, through this process, worked particularly well that you want to mention? And maybe any that you think you would like to improve on for next time?

00:15:40.62 >> Well, for the first time ever, I've been able to really be an active participant in working with the local level around a tough, challenging topic and feeling as though everybody was able to have a voice and get something that was quality to our communities around helping each other, through making better outcomes for our children.

00:16:08.75 >> I agree, Mary, having been working with you for a period of time for quite a long time. Typically, you've had to lead these groups, and I think having your voice as a part of it, because you're coming at it from ... Throughout the process, we talked about that there's the state perspective. There's the local perspective. There's the family perspective, and we wanted to make sure we captured all of that, and I think that this process allowed us to do that, and I don't think we would've been able to do that without working with IDC and having their support to bring these meetings together. I think that having ... I think I mentioned this before about having ... Basically, it was a four-part meeting. We were going to ... what our achievement, what the goal was. I think having it set up with the end in mind and created a plan to get there over the course of the four meetings was really important. I've been on groups before that have been going through the work and sometimes feels like it drags on. This process was ... Don't get me wrong. Mary mentioned it's tough work. We didn't always agree, but the point was is that we were able to work through things. People sometimes has to do some prework and do some follow-up in between, but there was a process with an end in mind, and we were able to meet that goal and were able to use this similar structure in the future.

00:17:47.78 >> And actually, Michelle and I continue to work on other projects such as: We're working with the National AEM Center for Accessibility of Educational Materials. That too is engaging a stakeholder group. I feel as though our knowledge, our experience and our confidence in being able to do a job that actually asks you to listen, listen, listen, be disappointed, be confused and yet go back to square one of why we actually are doing it, what do we really believe in? And moving the work beyond those barriers. It is very developmental in every process, and we've talked a long time around stakeholder and meeting together and convening for outcomes, but it really felt different this time for me, and it's made me look at things different with different lens.

00:18:46.53 >> Yeah, I agree completely with what both of you said. It felt so good when we were able to achieve that goal of: Okay, we've revised the survey based completely, really, on stakeholder input, and now we have this better-quality survey to move ahead with. So it was a really amazing process, and we achieved that goal at the end, which made it all worth it. And the work continues. In fact, what is coming next for you in this Indicator 8 world?

00:19:18.01 >> Well, as Michelle meets quite often with the Department of Education and the Bureau of Special Education, and not just around special education, but in general for families, and the fact of the matter is, I see her ability to be able to really have that equal voice. I see her confidence in being able to say, "Now that we've accomplished this, we need to make sure that" ... See, we've built some really nice technical assistance model. Now, we have to actually implement it across the department, across other organizations. We've always tried to do that, but it feels as though we've got something in place that's local, the state, and organizations feel as though they've built it together, and so what we're really looking at is, as we have actually talked about and listened to, we have recommendations for the department. We are responding to those recommendations, and the fact of the matter is, the department said, "Yes. You're right. We do have to engaged families, but we also have to make sure that within our work that we have a process in place that's built from the local level up to the state level." And so that's what we have first is. I think we were able to convince the department: take a step back, look at all the things under the hood, making sure all is compartmentalized to be able to make sure that it really does look like it's an entity that means business. Families are needing to be engaged, and Bureau of Department of Education really needs to continue the work.

00:21:14.93 >> And we are already ... So the four-part series was last year. The department has already engaged the stakeholder group again this year to look at the data, to look at why sharing the data is important to get feedback on ... We have a dashboard that displays the data statewide and per district, and so they've engaged the stakeholders, and we have all taken a look at that and provided some feedback. As we go into the spring for the next survey administration, the department has been soliciting feedback in terms of how we could market the survey, to look at representativeness, to look at survey response rate, and what do we need to do around education in the communities about what Indicator 8 is in terms of, essentially, why? Why would you want to respond to this survey? What is the point of completing it for families? Like the "What's in it for me?" So the fact that this stakeholder group was wanting to come back together to look at the data and to move this forward is a testament to what we had done last year and really helped spring us in to the next survey administration.

00:22:42.19 >> I really like what you said, Michelle, because I will be honest with you, and speaking with the commissioner and the leadership around the 603 survey, I was able to actually say, "In looking at your dashboard, we have comments from families and from schools that ... What we're seeing being displayed doesn't really make a lot of difference for us." It's not what we need. We need this. And so ... And for the first time, I really feel as though we actually asked folks about what they were seeing and actually had data people on the call to be able to realize, "Oh, what we thought was so shiny and nice and cool," meant nothing to families and schools. And that we needed to make the change. So in saying that, I think it's getting people to understand what we think we're engaging may not be. And what we need to do to get better at that and also feeling that everybody can have a voice, including the department, and so as we move forward, really want to make sure that we have the opportunity for our folks in the Bureau of Special Education, the folks in all of our title programs. Schools have the opportunity to go through this similar experience, and what the stakeholder group said is, yes, it's time to do that.

00:24:10.79 >> Great. A lot of good things, and this work is going to keep on going in different directions, but I really commend you all on how you were able to bring the stakeholders together and really meaningfully engage them and take their feedback and really incorporate it into this work, and I know you'll continue doing that moving forward.

00:24:31.93 >> And, Amy, I want to say that the way, the manner by which you were able to work with us made a difference, and I saw people really respond and respect that, yes, we have to look at tough things in front of us, and that we all could use some listening, and we call could change, and I would invite anyone, any state to really have that opportunity to do it, so we really appreciate the change you've made in New Hampshire.

00:25:10.42 >> Well, thank you for that. I appreciate it. It's been fun work. I really enjoy doing it and hope we can continue working together. Thank you, both, so much for being on and sharing your story of this Indicator 8 journey that we've been going on, and that will, I know, just keep on going. So thank you.

00:25:29.36 >> Thank you.

00:25:30.37 >> Thank you.

00:25:32.59 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###
00:00:01.52 >> You're listening to "A Date with Data," with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.78 >> So welcome to "A Date with Data." Today, I am joined by Julie Dean, the Early Childhood Special Education Inclusion Specialist, and Ryan Guzman, who is the 619 Coordinator, and both are with the Washington Office of the Superintendent of Public Instruction. We also have with us Kathy Linnehan, who is the Inclusionary Projects Coordinator with Educational Service District 105. And they are here today to share the story of their State Systemic Improvement Plan, or SSIP. And I want to start off, just if each of you could say a little bit about yourselves and how you work together. And, Julie, do you want to kick us off?

00:01:06.46 >> Sure. Hi, everyone. My name is Julie Dean, and, again, I am the Early Childhood Special Education Inclusion Specialist for the Washington Department of, really, Education. It's the Office of the Superintendent of Public Instruction, and I work primarily around the coaching and training within the early childhood work and space and also around data collection and analysis. So I'm really passionate about that work.

00:01:28.58 >> Thanks. Ryan?

00:01:30.90 >> Absolutely. Hello, everyone. My name is Ryan Guzman. I am, as we said earlier, the 619 Coordinator, also the SSIP, the State Systemic Improvement Plan, Lead for our state, so I've had the privilege to do that for a number of years, not only as a lead but also as a regional support to the work prior to coming in to our state agency. I come to you with, say, over 20 years of early childhood experience in our state-funded preschool programs as a self-contained preschool teacher as well as a early intervention provider, so it's a really unique opportunity to be able to use the different aspects of early childhood education in our State Systemic Improvement Plan, so we can look at that larger continuum, not only at 3 to 21 but really looking at how we're creating a continuum of life for the children and families that we serve.

00:02:22.39 >> Thanks. You have a really great perspective, having all of that experience and background. And, Kathy, how about you?

00:02:29.17 >> Thank you for having us today. I'm Kathy Linnehan, and I serve as Inclusionary Projects Coordinator at ESD 105. My work scans the gamut from preschool through high school, and I spent the majority of my teaching career in the classroom in kindergarten, but I do have an extensive background in ALE environment education, and I think that's where my passion came for this work.

00:02:53.66 >> Okay. Thank you. So I just want to start off by asking for you to share with us, what is your State Identified Measurable Result, or SIMR, and if you could give us just a brief overview of your SSIP?

00:03:07.85 >> Yeah. I'll take that. So our State Systemic Improvement Plan, when we had the opportunity to continue the work, I think one of the first steps we needed to take was to look at our SIMR.

00:03:18.78 >> Mm-hmm.

00:03:19.07 >> And in our past, so in the past iteration of the SSIP, what we did was that we had focused on early literacy, and what's unique to Washington state is that our early literacy work was focused on the intervention of early literacy within early childhood programs. I think a majority of the SSIPs that are being presented are typically in our K-12 system, and Washington took a unique perspective in starting at the beginning of our children's educational careers. So we thought that this would be a great opportunity to look at what we've been able to create within that first iteration and expand on that, so our SIMR for this round is a shift. So we went from early literacy, and we moved that focus over to social emotional learning. We wanted to make sure that we had a greater foundation for kids and providers and families within this space before we put that focus into academics. So we're looking at, what does intensive technical assistance in the area of social emotional learning do for a child when they enter a K-12 system? So we're utilizing our KEA, that's our Kindergarten Entry Assessment, in Washington. It's called WaKIDS, and we are using ... One of the six domains that our children are assessed in, social emotional learning, as a metric to measure growth over time.

00:04:42.71 >> I'm interested in knowing, like you said, Ryan, it's a little unusual. Not many states are looking at early childhood in their SSIP. Why did you all decide to go that route and have that be your focus?

00:04:56.00 >> That's a great question. Based upon our data, it's the wondering. We ...

00:05:01.26 >> Because your data are pretty good?

00:05:03.32 >> Again, great question.

00:05:05.11 >> Sorry.

00:05:06.35 >> It's not! What happened was, when we started to look back, especially when we went into this next cycle, within the work that we do with our state design team, we're very interactive and intentionally are collaborating at every level that we work, so bringing the data from our performance indicators to our state design team as well as our SEAC, our Special Education Advisory Council, to say, "Here's what our focus is in. Are we in the right space?" And based upon feedback, they said, "You know where we need to prioritize our work as we move forward? It's going to be around early childhood special education," specifically within our Indicator B6 and looking at our least restrictive environment. So, again, when we looked at our data, it was a point of urgency for us to really lean into what was happening within our early childhood programs and how we could better serve and support our children. When we step back and look beyond early childhood performance indicators around least restrictive environment, then we start to look at our performance indicators around state assessment and student outcomes, and we can look at discipline, and when we start to really lean in to those areas in our K-12 system, there was a lot of questions about, "But what about early childhood?" And we didn't, in some cases, have data to answer those questions, so it created, again, additional urgency to say, "What are we missing here and why?" So it was, again, the beginning of a different conversation but a need for us to step in and to say: We really need to focus on not only early childhood programming in our state. We need to address the disparities that we are having in early childhood as opposed to our K-12 systems. We had added funding within our legislature to support inclusionary practices, professional development work in our K-12 systems, and we're very blessed to get about $25 million over a 3-year span to support K-12 implementation of inclusionary practices to build cohorts and training opportunities. What was missing was P. Where's the preschool?

00:07:12.42 >> Mm-hmm.

00:07:12.81 >> And knowing that IDEA is in 3-to-21 requirement for school districts in Part B, and, yeah, we really wanted to make sure that we were fully encompassing the opportunities for all of our partners or practitioners in this work, and we knew that we weren't doing that to the best of our abilities, and this was our opportunity to lean in to the work, to look at the data, to bring it to our state leads, and then Kathy, who's with us today, is one of our incredible implementation specialists who has been able to create a network of regional partners, our school district's inclusion champion. So we have preschool included in the champions. And so she's leading that network at the regional levels so that we can model up what we need to do and model down. So how are we really ... Being who we need to see, our state needs to look at the data to interpret and to tell and to understand the story and to be okay with where we have our deficits and gaps so that we can change and strategize with our implementation specialists at the regional level, so we can get that information, those strategies and best practices down to our local teams.

00:08:24.12 >> Yeah. That's fantastic. That's lots of great reasons to select that as your focus. You saw gaps and stakeholder-driven ... what they thought was needed and, really, that missing piece of the preschool to make sure they're not forgotten and just focus on them.

00:08:44.35 >> And I can add too, in terms of the data, we'd been utilizing the state benchmark of quality tool, which is a self-assessment tool that our state design team, through the State Systemic Improvement Plan, and prior iterations around Washington Improvement Model implementation, we utilized that tool similarly to the way that Kathy utilizes that tool with participating programs, and we were able to identify significant increase in professional development and training in those components after really shifting towards these alignment, this alignment practice with MPFS and developing this infrastructure of coaches, we saw some significant gains in the data specific to those areas, and further conversation around other components within that benchmark of quality, like cultivating a culture of data-informed decision making and really employing a system-wide data-collection system, which we now were able to deploy this last year. So those conversations that Ryan had mentioned are all based in really clear data collection and ongoing conversation based on that data.

00:09:45.52 >> Mm-hmm. And it sounds like you did, in terms of changing from what you were doing kind of under the first part of the SSIP moving into the FFY 2020 and 2025 package, you kind of shifted from early literacy to social emotional. Can you talk a little bit about that shift and maybe other changes, some lessons you learned from the focus on early literacy that ... things you might've improved on differently or continue doing that were working well in this new iteration?

00:10:16.24 >> What we knew in the past with our KEA, Kindergarten Entry Assessment, was that our children with disabilities were consistently performing at about a 30 percent achievement gap when compared to their nondisabled peers.

00:10:29.71 >> Mm-hmm.

00:10:30.19 >> And that was consistent over the course of time, and even if you think about the intersect of race, ethnicity, homelessness, our migrant population, we're seeing, again, that achievement gap. So we noticed, again, where we needed to address that, but what we've received from our regional leads was, we can't get to academics if we haven't addressed the child, the environment, the community that they come from, and really addressing the whole child first and then getting to that place of academics and academic achievement.

00:11:04.66 >> Mm-hmm.

00:11:05.06 >> If you can address the whole child, most like, you're creating a space where that child feels safe and feels like they can trust the adults in those environments, and they're more open and ready to be learners. It also, and we know, helps with that transition, and when they move from a preschool to a kindergarten or a different setting, so those are some of the things that we took into consideration.

00:11:29.88 >> Mm-hmm.

00:11:31.14 >> Again, also, our LRE data, so looking at our data and knowing that in Washington state, we were not at a level that we wanted to be at when it came to creating and maintaining access points for children with disabilities. And I think most importantly, and I'll say this once, and I know some of our friends have heard this before, but in Washington state, we don't use the term stakeholder. We use the term community partner or partners. So we do that because we want to ensure that we're being respectful and mindful of our tribal members, our indigenous communities and that they are a sovereign nation and not as different than those who that are not within that space. So that is, moving forward, just something that we have decided collectively within our agency to use as a term.

00:12:17.88 >> Mm-hmm.

00:12:18.32 >> And with that, that was based upon community feedback, so based upon the feedback of our partners, we really wanted to make sure that we were shifting our focus to ensure the voice of community partners and families at the front of everything we did.

00:12:34.29 >> And is there something that you're really proud of in Washington related to your SSIP, something you want to highlight that has really worked well that other SEAs and districts might be interested in hearing about?

00:12:49.19 >> I think Ryan said it so beautifully too, this idea of establishing and aligning practices and systems, and I think that that is one of the biggest celebrations that we have is really looking at a lining of a network in terms of our coaching and training practices to really complement and work within this complex mixed-delivery system and also with our K-12 partners. I think we've established a really robust and really passionate data community, community of practice, and we've been able to do that because we've established a data submission platform. Through the Pyramid Model Consortium, we've been able to contract and utilize the Pyramid Model Implementation Database System. That system has given us some flexibility and how we submit data, how we analyze data, and frankly, there's been a pretty significant increase in data submission across our sites. As Ryan had mentioned, starting in 2020 to 2021, there were 11 SSIP, S-S-I-P, sites, and we've accelerated up to 26, and now we're recruiting for the next year, doing that recruitment. What happened? We also have participating sites across the state that are not SSIP sites but are engaging in this database system, and that's expanded over to across 131 different LEAs and programs.

00:14:09.00 >> Wow.

00:14:09.54 >> So this has given us an opportunity to really identify that district capacity assessment and utilize the early childhood benchmark of quality to kind of pinpoint critical elements within a team and really going back to systems and infrastructures, how can we support those teams that continue to promote [Indistinct] and also have conversations about data collection? What is our data telling us?

00:14:34.17 >> Mm-hmm.

00:14:34.41 >> We're starting to have deep conversations about our statewide behavior incident reporting data, which is data that's cross-analyzed with race, ethnicity, gender, IEP and DLL status, and so these are really the rich conversations that are deeply rooted in data analysis, and I think Ryan had mentioned this too, as we continue to focus on social emotional supports, how are we ensuring that we're not weaponizing the data that we're collecting? And that there aren't ... What are our focal group? And is there disproportionate data that we're collecting, and how do we then provide that rigorous technical assistance and support that is needed to engage in those deeper conversation?

00:15:18.04 >> Kathy, what about you?

00:15:19.83 >> Okay. So I have a unique position in that I work half of my time in the ECSE world and half of my time in the K-12 world, so I'm bridging that gap, and what I have always said is that the preschool has led the way in this work. With the ECSE program, we have teachers throughout our regions, I'm hearing feedback that are saying from high school educators, why are we not involved in this? So this is kind of ... We're seeing this roll together and become this wonderful framework for our state. I would say if I'm thinking about ... One of the successes of this program has been the enthusiasm. As I said, it's a big bite for districts and teams to take on, but when they do, and they get deeply involved in this work, the motivation that they have to grow this program is exceptional. It's outstanding work that they're working in. What they see is that even if they have systems in place that are doing some of this work, they're siloed.

00:16:20.22 >> Mm-hmm.

00:16:20.63 >> And so this program is really bringing those together, so that data is in one system in one place for districts to easily access and to analyze and to look deeply into what they thought they need and what's really true. And I think about the decisions that they're making with this data is far-reaching. They're thinking about, "What's the PD that our teachers need? Which teachers need additional coaching within those classroom environments? What policies, what procedures do we need to think about that we need to maybe change, edit, adjust or add to?" And so all of this brings this enthusiasm to these teams, and we're seeing that the word is getting out, and people are ...

00:17:04.41 >> Excited.

00:17:04.88 >> ... coming to us now.

00:17:07.26 >> So now, what do you have in the works? What plans do you have? What is the future of your SSIP?

00:17:14.27 >> Everything is in the works. I think ... You hit it. There's a lot of connections, and I think that's the most important part. Special education is the supplementary supporter. Special education can lead the charge, but we're not going to do this alone. And I think that's one of the biggest things that we have been able to do with this is really pull in a number of people, partners, agencies across our division within our own agency but also cross-sector partners that are really like, "What's going on? I want to be a part of this," and that's ... I would say, one of the more exciting parts of this is, in Washington state, within our early childhood programs, we have a mixed-delivery system. We don't have universal preschool, and I think for many of those that might be listening today, they're in the same space where you're wondering ... Funding early childhood programming is very difficult. It's very complex, and it is something that we're always seeking more information on.

00:18:08.16 >> It excites me too, just to jump in to say ... This is Julie, and I wanted to ... I was making notes as Ryan was talking because I think, and also Kathy, this idea of, what does fidelity mean? And knowing that this data collection can give us a better understanding of what fidelity means and whom and under what context, and I think this work is really linked back to implementations, buy-ins and understanding that there's different stages of implementation, and so fidelity looks different based on different program need, and I think that that speaks again to this being responsive to our community partners, and I think for us this past year, we've really focused on criteria around training, and what does training look like within this professional development community? And now, we're engaging in deeper conversations around coaching and employing the features of practice-based coaching and also transformational coaching to really examine and explore the impacts of intergenerational trauma within our tribal communities in Washington state and Black communities and really looking at our data to examine how we can do better. And how are we then being responsive to this data in a timely way? And so that's what I'm hoping that we'll continue to do is engage in cultivating a community, a roundtable, a data carousel, where we can really look at our data regularly, and that can inform our current practice and that shared-system sort of driven leadership that we see in our MTSS infrastructure. So I think that that's that focus, our social emotional development, professional development community, we require ongoing coaching and support and deep, deep analysis and data to support where to coach and when to coach and why to coach. And to whom are we coaching? And then who's coaching our coaches?

00:19:58.23 >> Kathy, what would you like to add?

00:20:00.61 >> Well, I think the implementation science is really important, so as our teams are moving forward, it takes time. It takes time to do this work, and what we're seeing is, they're digging so much deeper into this than any other initiative that we've seen. This is quite a commitment for our state to make to move this work forward. I see that people are coming to us, asking us to be part of this work, and I think that that's going to continue to grow so that we have that high-quality environment for all students in preschool or across our state.

00:20:32.96 >> Again, thank you so much for sharing your story. So much going on and so much to come, and just really appreciate you taking the time out. Thank you for being on.

00:20:45.31 >> Absolutely. Thank you for having us.

00:20:47.24 >> Thank you.

00:20:48.41 >> Thank you!

00:20:50.43 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the Podcast page on the IDC website at ideadata.org.

View Details

Resources:SEA Data Processes Toolkit | IDC - IDEA Data Center
Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.49 >> Hello! Thank you for joining this episode of "A Date with Data." I am chatting with Jamie Chambers, Part B Data Manager, and Kelly Wojcik, Director of Accountability. Both are with the Nebraska Department of Education. Jamie, Kelly, as well as others with the Nebraska Department of Education have been busily completing IDC's data processes over the last few months, and they're going to share with us what that has been like so far. For those of you who aren't familiar with the data processes, it's a collection of templates that states can use to document all of their state-level IDEA data collection reporting, procedures and activities, so please check them out on our IDC website. Welcome, Jamie and Kelly. Can you each just tell us a little bit about yourselves and your background? And, Kelly, do you want to start off?

00:01:17.20 >> I'll go ahead and start off. I am, as was announced, Kelly Wojcik. I'm currently with the Nebraska Department of Education as the Director of Accountability. I was a classroom teacher for 16 years working primarily with students who were deaf and hard-of-hearing and then moved into the state role in Illinois and then landed in the state department at NDE. As part of my responsibilities, I do a lot of work with data and have been with the State of Nebraska now for 8 years, going on 9 years.

00:01:54.62 >> Time flies by.

00:01:56.30 >> It does.

00:01:57.62 >> Thanks, Kelly. How about you, Jamie?

00:02:00.61 >> As you mentioned, my name is Jamie Chambers. I'm the Part B Data Manager here in Nebraska, and I am approaching a year of being in this role, but I've been with the state for about 3 years now. I started off on the Part B team for ages 3 through 21 which was a huge shift from my previous position as ... I call myself a reformed preschool teacher. I taught in preschool and Head Starts for 12 years prior to coming to the state, so I'm still not exactly sure how I ended up on the school-age side of special education or how I ended up as the data manager, but I'm really excited that I'm here and excited to be able to bring that program perspective to the data world.

00:02:51.30 >> Great. All right. So let's dive in. Can you talk a little bit about why you decided to engage in this data process work?

00:03:03.73 >> I'll go ahead and start in.

00:03:06.45 >> Okay.

00:03:07.28 >> We actually started the data processes work about 5 years ago, and unfortunately at that time, we were kind of in the same position that we are now. We had just switched data managers, and we were questioning how things were done, why they were done that way, but we never did finish and get through all of the data processes.

00:03:31.44 >> And for me, as a new data manager, I also have a new EDFacts Coordinator, and then it kind of just seemed like everyone I was working with was kind of new to their position. And we do have a NEDA B contract, and through that I've had the opportunity to be mentored by Nancy Johnson, and with my conversations with her and her hearing some of the struggles that I was having, she highly suggested going through these data processes. And we are also a Cohort 2 state, and so during the Interactive Institute, we heard how critical it was and helpful these data processes would be in order to prepare for the DMS 2.0, and that just expedited our decision to get these scheduled.

00:04:16.52 >> Yeah, definitely. So much of that process, the DMS process, is providing evidence and documentation of your processes and procedures. So how did you prepare for the data processes work? It involves IDC coming in and helping to facilitate and really spending days, hours, weeks together really going through very carefully step by step exactly what you all at the state are doing from start to finish in each of the different 618 collections and SPP/APR indicators. So what are some of the things you did to get ready for that work?

00:04:59.69 >> Really for us to prepare, we started with the IDC staff, and we met with those that would be supporting us getting through these processes to really discuss and have a good understanding of who the essential people were that we needed to get together, and then we worked on creating that agenda for the first data collection we decided to tackle.

00:05:23.06 >> Yeah, they helped us identify any of the essential people that we wanted to have at the table, and then I've started reaching out to begin scheduling, so that those people could attend. Then we decided to encourage in-person attendance but also offer a hybrid option as a last resort to keep everyone involved in those conversations, and I will say being the one that had the conversations to get that schedule going, it's not for the faint of heart. You really have to be persistent and keep heading towards that goal to be able to accomplish it.

00:06:02.79 >> Yeah, and were you involving folks even beyond your special ed group, going into people from Assessment and from IT and ...

00:06:11.99 >> Oh, yes.

00:06:12.56 >> Yeah, pulling kind of just beyond your team which was ...

00:06:15.79 >> Anyone and everyone we can involve, yes.

00:06:18.11 >> Yes, for sure. So how has the experience been so far? How many data processes have you completed, and just what has it been like?

00:06:30.98 >> Well, so far we've completed one, and even with that one that's been completed, I think we already have some opportunities to go back and tweak some language and to review that information, but we'll get to that a little bit later. In terms of the experience so far and having done it once before, it has been such an incredible professional development activity for everyone who's been involved. The facilitators who have been helping us through that process recently have had such high-level quality questions that really have pushed the conversation and has really allowed the team to go into so much more depth of the processes than we've ever had before which has really helped our understanding become much richer than we ever got to the first time we went through it. So just to have that depth of understanding and to understand all of the little quirks and nuances that happen with the data has just been indescribable.

00:07:45.88 >> And for me, my reformed preschool teacher is going to be coming out here. The thing that has been the most helpful in this process to me is to create and build relationships because that fosters better communication between all of the parties that work with the data. I personally learn best through conversations, and so these relationships are essential for me to be successful in this position. I don't do well with just being able to read something and feel confident in moving forward as if I know what's going on. I need to have these conversations with the key people that are involved to know who to ask questions to and how to move forward in the work.

00:08:29.37 >> Yeah, I think from my experience being in that facilitator role, it's not just about getting the documentation done. That's a big piece of it, but it's the process itself and breaking down some of those silos and building those relationships, like you said, and especially as a new data manager, I would imagine for you, Jamie, being able to meet people and actually sit down in person with people that you probably otherwise wouldn't have had the opportunity to do that with, if not for having to do this process.

00:09:03.26 >> Yes, exactly.

00:09:05.74 >> Are there any tips or lessons learned that you could share with other states that are just starting to document their data processes?

00:09:16.36 >> My first is, you've got to commit.

00:09:19.14 >> Mmm.

00:09:20.97 >> I think where we went wrong the first time around was that there were really only one or two of us that really wanted to do this process, and I think that's why it never finished. And we've had the opportunity to talk to other states who have also engaged in this process, and one of those states said that they had scheduled all of the dates for all of the protocols at the very beginning, and looking back, I really wish we would have done that because it's taking us ... We had our first one earlier in the fall, and it took us 1 to 2 months just to schedule the next collection. So part of me being an eager beaver, I want to get it all done because it has been such a rich experience and that having to wait for a couple of months to get into it again is really hard.

00:10:21.34 >> Mm-hmm.

00:10:22.83 >> Because the conversations that emerged while we were working on it were just so ... I hadn't realized how much I didn't understand.

00:10:34.80 >> Right.

00:10:35.71 >> So it's just for ... Waiting for that next collection is difficult at best.

00:10:45.47 >> I agree, and my advice would be just do it. It is super overwhelming at first, but the conversations and the time together building those relationships with the people who are doing this work is invaluable. I really can't imagine trying to do this job without working through these processes because it's just so overwhelming, all of the information and the deadlines and even working through ... Kelly mentioned we've only made it through one collection so far, but working through that one collection ... We did Child Count to start ... has already helped when I was preparing for all of the November 2nd collections. Even though we haven't done those data processes yet, doing the Child Count collection helped me through that submission, and I'm excited to be able to get to those collections during our data processes. And actually, some of my questions that have come up through those collections are, what's helping us know which collection we should do next? The next one we have scheduled is the assessment collection, but then we already know after that we want to go right into the Discipline collection because of questions that I had during that submission process.

00:12:05.20 >> Yeah, and you mentioned how you've done ... It was in person or a hybrid, that one that you've done so far in terms of ...

00:12:14.76 >> Yeah.

00:12:15.26 >> Yeah.

00:12:15.63 >> For the Child Count submission, we did ... We strongly encouraged in-person attendance. We didn't actually tell people that there was going to be a virtual option.

00:12:26.06 >> Right.

00:12:26.50 >> But we did offer that virtual option because we had some of our NEDA B partners that we wanted to have involved in the conversation, and they were joining virtually. In fact, our EDFacts Coordinator even made the comment, once she was there in person, that had she realized there was a virtual option, she probably would have chosen that option, but by the end of the day, seeing the value of those in-person conversations, even she was saying for our next collection that we're doing an assessment, she was encouraging us to do the same thing, to try to get the key people in person at the table, but then if there were other people, or if there's no other option, to have them on virtually.

00:13:15.13 >> Okay, good. So it seemed to work, sort of having that somewhat hybrid?

00:13:19.71 >> It did work to have it have that hybrid option, but I was one of the individuals who had part of a day in person and then did follow-up with the virtual option, and it truly was not the same. It was a much richer experience in person because you could see, from the people in the room, the confusion which then prompted the questions to help clarify and ensure that everybody truly understood what was said and how that linked to what was happening. So although I appreciated having the hybrid option, and I needed it at that point in time, it was a much richer experience in person.

00:14:13.52 >> Yeah, that makes sense, really being able to see people's facial expressions up close and those kind of other ways of communication, just kind of getting a sense of how people are doing and what they're thinking doesn't always come through clearly virtually.

00:14:30.30 >> And what was super helpful for me is I was sitting right next to our EDFacts Coordinator, and so in a virtual setting, we wouldn't have been able to have some of the sidebar conversations or questions or just reading each other's body language to get through the information like we had when we were in person. It would not have been such a rich discussion had one or both of us been virtual. Having us both in person was crucial.

00:15:01.30 >> Mm-hmm. What do you hope will be kind of the ultimate outcome, or what do you hope will come out of going through all these data processes together with the state? What are kind of your hopes and dreams for it?

00:15:17.18 >> My hopes and dreams for this really is that everybody on the team have a basic understanding of the processes and what it takes to get the data that we need. We've, for a while now, been fighting against the, "Well, this is the way it's always been done," and just because that's the way things were done in the past doesn't mean that, A, it's the right way, or B, that, huh, OSEP changed the way we collect that. So I'm glad that's the way we did it, but it doesn't apply now.

00:15:56.87 >> Mm-hmm. Things have changed.

00:16:00.19 >> Things have changed, so I'm hoping that we can eliminate the conversations of, "Well, that's the way it's always been done," because it's irrelevant, and my goal last time when we did this was to pull it off of the shelf and update it on an annual basis.

00:16:21.80 >> Yeah.

00:16:22.02 >> And I think what we've seen already is the importance of continually updating it because people change, processes change. Even if people remain the same, it's getting the cobwebs out of their brains to make sure that we are doing it accurately.

00:16:43.87 >> And I really hope to be able to get to a place where I'm being more proactive instead of reactive, and not just me, but as ... all of us, as a team, to all of these deadlines and submissions. I would love for us all to have the confidence in the data that we're submitting including the program people. I want them to feel confident in what they see, in the data matching the conversations that they're having with districts. I want to make sure that the entire team understands what the expectations are and what the district needs are during each submission so that, when districts have questions, any of us are prepared to answer those questions consistently.

00:17:29.92 >> So everyone is on the same page and has the same kind of knowledge base, and I think your point, too, Kelly, that it's a living and breathing document. It's something that when there are changes to your process procedures or OSEP makes changes, pull it out and update it so that it's always sort of current and useful.

00:17:52.61 >> Yeah, and I'm sure this doesn't happen to anyone else in any other state, but we sometimes have districts who like to fish for answers where, if they don't get the answer that they want from the first person that they call, they try calling another person, and we try to circumvent that with ... We have a call log where we track all of the phone calls, so that when we get a phone call or a question, we can double check to see if they've already talked to someone else on our team, and that's great, but it would also be great if we all had that consistency in our answers so that we don't have to worry about catching people fishing for answers.

00:18:34.01 >> Yes, consistency for sure, and accuracy.

00:18:38.45 >> Well, and I think along with that is being able to share the wealth of responsibility.

00:18:45.98 >> Mm-hmm.

00:18:47.31 >> I'm, quote-unquote, "one of the old-timers," and I'm just in my ninth year. Jamie is our third data manager with Nebraska since I've been here. She's, by far, my favorite, and I hope she knows that.

00:19:02.33 >> I didn't dare to say that.

00:19:04.72 >> But being a data manager, there's a lot of responsibility with that. It's a great deal of work, and our districts have numerous questions. And when Jamie came on as the Accountability Director, I didn't have what Jamie needed to help her understand what those responsibilities were. She'd ask questions, and I'm like, "Well, this is what the end product should be," but I had no way to guide her in terms of what it took to get to that place.

00:19:39.68 >> Mm-hmm.

00:19:41.48 >> And knowing how much we get inundated with questions from our districts, those shouldn't all have to go to Jamie. Jamie shouldn't be the only holder of all of that information to be able to answer those questions. Every individual on our staff should have a base level of understanding, so we can triage, and when a submission is due, we can take that stress off of Jamie and only bring the ones to her that have gotten to a level of depth that that basic understanding can't help answer the question for the person who's bringing it to the table. And with that, hopefully Jamie won't be overwhelmed and leave us.

00:20:30.04 >> Right!

00:20:30.31 >> So I'm going to be very selfish on that count.

00:20:34.69 >> Yeah, with the high turnover, it's ... This process serves many purposes, hopefully to keep people, but if you do happen to have new folks in, well, here's a guide and a road map to how to do all of this.

00:20:46.85 >> Indeed.

00:20:48.42 >> And that's where I'm going to go back and rely on my preschool teacher experience because being a data manager is not just about figuring out how to do that work, but like Kelly said, it's about making sure other people are understanding the work. And I always kind of see myself as this bridge between data and program, and since I have the experience on the program side, I have to up my knowledge of the data side to help create this bridge for everyone else. And so it's important for each data manager to know whichever side your strength is on, then you just are going to have to focus on the other side to help create that bridge for everyone else, so that everyone can be on the same page and share the responsibilities.

00:21:41.39 >> Anything else either of you want to add about working through these data processes?

00:21:47.03 >> Working through those processes, the goal is that we'll be able to better use our data.

00:21:52.41 >> Mm-hmm.

00:21:53.14 >> I've been a huge proponent of if we have the data, we should be using it, and luckily I have like-minded people, like Jamie, who understand the data, understand the impact it has on our programs, and I'm hoping that as we are more familiar with the data processes themselves, that will just enhance our ability to effectively use that data in our decision-making processes which impact our monitoring. It impacts our general supervision overall, and ultimately it impacts the lives of our kids and improves their outcomes. So this is one of the stepping stones to make sure that our students are achieving the outcomes we anticipate they can achieve.

00:22:42.92 >> Yes.

00:22:43.14 >> And that's always my North Star is improving outcomes for students, and at the end of the day, everything that we do is for children.

00:22:53.40 >> Yes, and this is, like you said, one of those steps that on the surface may not look like it but definitely leads ... is a necessary step to get to that point of ultimately improving services and outcomes.

00:23:09.16 >> Mm-hmm.

00:23:10.36 >> Well, thank you, both, so much for being on. This was great.

00:23:14.65 >> It was our pleasure. Thanks for having us.

00:23:17.46 >> Yeah, thank you so much.

00:23:20.29 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org.

View Details

Reach out to us if you want to access Podcast resources, submit questions related to episodes, or share ideas for future topics. We’d love to hear from you!

You can contact us via the Podcast page on the IDC website at https://ideadata.org/.

### Episode Transcript ###00:00:01.52 >> You're listening to "A Date With Data" with your host, Amy Bitterman.

00:00:19.50 >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.91 >> Welcome to today's episode of "A Date With Data." I am so fortunate to be continuing to explore the topic of what it means to be a data quality influencer in the world of IDEA data. And I'm thrilled to be joined by Dr. Herbert Boat, Assistant Director, and Jessica Tu, Data Manager, both from the American Samoa Department of Education Special Education Division. So, welcome.

00:00:52.43 >> Mm-hmm.

00:00:53.37 >> Let's start out. If you could just say a little bit about who you are and your role with the SEA ...

00:00:59.58 >> All right. Hello, and [foreign]. I'm Herbert Boat Jr. from American Samoa. I'm the Assistant Director for Special Education under the umbrella of the Department of Education here in American Samoa. And I have served as an AD, Assistant Director, for about 2 years now. I've been a ... teacher for, I'll say, 13 years, and I became a program director for the same division for 5 years. So as of now, to this year, marks 3 years of service ...

00:01:38.60 >> Wow.

00:01:39.00 >> ... with the Department of Education.

00:01:40.80 >> Oh, my gosh. And, yeah, it's an honor to be on this program on this podcast, so hopefully we'll do good with this interview, and hope to see you guys sometime soon. Thank you.

00:01:52.75 >> Thanks. How about you, Jessica?

00:01:55.39 >> Hi. My name is Jessica Tu. I'm the Part B Data Manager for Special Education Division, again, under the American Samoa Department of Education. My whole role and responsibility is to make sure that students with disabilities are accounted for in the student count for ASDOE. And as of this year, we're looking at 14,000-plus students within ASDOE, and we have about 535 students with disabilities, give or take. We have some transferring, some coming in. And like our assistant director, I have also started off working as a special education teacher since 2006, and now I work as an ES supervisory position to oversee the program in two high schools, and now I am actually working data to make sure that the data is of quality and that we share it with our department, our inner agencies as well as our public.

00:02:53.44 >> Great, so you both have had varied experiences in the world of special education and data and been in the field and been for so long at the Department of Ed, so you have a lot of experiences and knowledge, I'm sure, to share with us. Can you talk a little bit about how you feel like you influence the quality of American Samoa's SEA data just with so much going on, DMS and getting ready for the next SPP/APR, changes to EDFacts reporting, all of it? How do you feel in your role that you're influencing the quality of your data?

00:03:32.87 >> Part of my responsibility as a Part B Data Manager is also being part of our subgroups within our Department of Education. I am also part of the Data Governance Committee and also the Accountability Committee, and for me to go back into these committees and share with them the data that we have from special ed, the services that are needed and what assessments look like for students with disabilities, we're able to come back to the drawing board. And our team lead for the subcommittees is able to come back to the committee that our AD is part of, which is the leadership team for ASDOE, and to share and say, "Hey, just because we're a whole DOE doesn't mean we can't cater to our students that are actually struggling or actually in need of that service." And so part of this data quality that we keep talking about, I would like to echo what our compliance officers ... Over the years, it was always results, results, results, compliance, compliance, compliance. But what are we actually doing? And so for the past 3 years under the supervision of AD Boat, Dr. Boat, and our previous assistant director, we've been able to come in and have a voice within the subcommittees, being able to be part of these committees and make decisions for ASDOE as a whole. And so if you see the programs that are going on now with feedback from our AD, but then the leadership team, you see the whole of DOE coming together. It's full inclusion, and the Director for DOE wants to make sure that everybody is fully included even though you're from these interagencies within DOE.

00:05:12.73 >> I just want to add on that what just was mentioned about the inner agencies and the support from our director and deputies. Yeah, it's been at full support from them, and we've been meeting as leadership, and our director and deputies always support what we have to come up with. It's all for the students, the academics and especially our students with disabilities.

00:05:37.07 >> Wow. That's great, so trying to break down those silos and really move beyond just streamlined focusing, it sounds like, on just students with disabilities or students without but bridging those gaps and really trying to full impact across the board. So that's ...

00:05:55.71 >> Exactly.

00:05:56.01 >> That's amazing.

00:05:58.29 >> Also, Amy ...

00:06:00.44 >> Yeah.

00:06:00.68 >> With the preparation with our DMS decision next year, yeah, so now we've been meeting with our state lead, Dr. Rhonda Santino and our leaders here at our state office, even our data manager, which is, it's getting there. But we still need a lot of data and a lot of documentations that we need to come up before the reasons come up. And for our data manager, yeah, I've seen the way she works. She's under my supervision, and it's a lot of work, and she still does school visits to make sure the data is on top with the ESes and make sure that the data matches our IDEA system.

00:06:46.55 >> Yeah, the DMS, I know, is a big undertaking for everybody, and you still have to do your regular job and everything you've been doing ...

00:06:54.27 >> Right.

00:06:55.27 >> ... apart from that and then adding that on top of it, so I really commend you, and I know that it can be tough. But it sounds like you're getting ... You're laying the groundwork and doing a lot of preparation so that I'm sure when the time comes, you'll be ready.

00:07:11.15 >> Right, just keeping our fingers crossed.

00:07:14.66 >> Yes.

00:07:15.29 >> [Indistinct].

00:07:16.71 >> Yeah, I know you still have some time, and that's great, that you have that. And having these conversations, ongoing conversations, with Department of Ed, too, must be really helping to kind of set the expectations and get everyone on the same page. Have you found the tools that I know are available, the templates, those lists of questions they're going to ask and the documentation and evidence? Is that sort of what you're following to gather information, or are you doing something different?

00:07:47.88 >> Exactly.

00:07:48.39 >> Yeah.

00:07:48.52 >> That's what they're trying to help us with, yeah, with what they're going to be looking at and what we need to be prepared for.

00:07:54.93 >> Yeah.

00:07:55.02 >> So as of right now, we still have a lot of work to do.

00:07:59.25 >> Yeah, your [Indistinct] from all SCAs that are going through it. You're not alone, and so it's good. Some are even, I know, deep in the process right now with having their site visits and those sorts of things. So we at IDC at least are going to try to learn as much as we can from them and be able to share out with other SCAs like you all who maybe have a little bit more time, so hopefully that will also kind of help everybody with this. Can you talk about any challenges that you encounter being a data quality influencer and what you do to try to address them?

00:08:32.80 >> Part of the challenge is trying to be consistent with the data input from the school because I just oversee the system from a special education point of view.

00:08:42.55 >> Yeah.

00:08:43.06 >> And we still have our mother department to go through, which is our IDS office, Integrated Data System office.

00:08:49.66 >> Okay.

00:08:49.91 >> And another challenge is, because the SIS, which is the Student Informational System, it's fairly new to DOE right now, before, it used to be SMS. So the educational specialists, which is the ESes, are slowly now beginning to get the gist of the features of the system and how detailed it is because actually now we're able to go in. Do the input. Print out a draft. Carry it to the meeting. Come back. Make the changes. Finalize it, and then print it out for a final folder. Previous years, it was a little bit more. We had to come write it. Go to the meeting. Do the input. Upload files, so it was like a ... working backwards, but now that's the major challenge, though, is to make sure that whatever information is going on the system is consistent with the hard copy file and to make sure that the numbers match what we have in the system with the count we have out in the schools because we currently cater to 29 schools, which is six high schools and 23 elementary schools, but we're not just in one major island. We have four schools on the outer islands, and for us, compared to stateside, we're considered a regional district. We're not broken up into individual districts ...

00:10:11.03 >> Mm-hmm.

00:10:11.24 >> ... like it is stateside.

00:10:12.50 >> Mm-hmm.

00:10:13.37 >> That's another challenge for us as data quality influencers, because sometimes when they come out to our meetings, our data collaborative meetings that we have every month, hearing them talk about how it is, well, in this district, we have this, and we have to go through a superintendent. And for me, I would like to be in the meeting how we had our Pacific collaborative where we all relate to one another because we're just all one district.

00:10:37.02 >> Yeah.

00:10:37.52 >> We oversee ... rules and we look at, so that's kind of another thing. But all in all, it's good being the Part B Data Manager, and again, being part of the subcommittees has been a positive grasp for us here at Special Education.

00:10:54.26 >> Yeah, and to hear and just be more in the know of what's going on and when changes are happening and what's coming up and to be able to, like you said, have your voice heard through all of it, too, that seems really key, for sure.

00:11:07.46 >> Can I add on?

00:11:09.89 >> Of course.

00:11:10.57 >> One of the challenges is the RE/SSR Educational Specialists out in the schools.

00:11:16.72 >> Okay.

00:11:17.00 >> It's that inputting the data on the system because most of them ... They wait for the last minute.

00:11:23.23 >> Mm-hmm.

00:11:23.47 >> So just, yeah, e-mailing them, making sure, and she also does the visit out in schools to make sure that they match the system and what they have on file. Yeah, so she's always on top of that, trying to make sure because I know. I even see the e-mails myself because I see most of the e-mails, and that's one of the challenges, is the ESes and teachers out in the schools. And, plus, also with the new features with the IDS service, the data service online right now is ... Everything is on there concerning the IEP files and stuff. But it's just that I guess through the time of the COVID, most of the instructions was put on hold.

00:12:11.79 >> Mm-hmm.

00:12:12.23 >> So all the teachers that ESes now is trying to get back on top of their data system and make sure that they enter everything on the system, so that's one of the challenges, I guess because during the COVID shutdown and also with the Internet connection.

00:12:28.31 >> Mm-hmm.

00:12:28.87 >> Yeah, like Jessica was saying, we have four schools in our neighboring islands.

00:12:33.82 >> Mm-hmm.

00:12:34.03 >> So if the Internet is down, it's hard for them to connect with us.

00:12:37.99 >> Yeah.

00:12:38.09 >> And they even have a hard time to go on the system to update their data.

00:12:42.83 >> Wow, and also maybe is there still kind of a learning curve? You said the SIS ... It's a newer system ...

00:12:49.75 >> Correct.

00:12:49.91 >> ... maybe for them and still kind of trying to work out the kinks and ...

00:12:52.74 >> Yeah.

00:12:53.15 >> So it sounds like, yeah, there's a lot going on. But it sounds like you're doing a lot of really personalized TA and really walking through individually and doing all of that great checking. Is it student-level files, even, that you're getting down to where you're making sure that that, like you said, is matching what they're entering in the system? That's pretty down-in-the-weeds there with them, so ... And are you traveling out to the four schools, too, that are off the island more?

00:13:23.39 >> Yeah, going out the ending of this month, right after Thanksgiving ...

00:13:27.17 >> Yeah.

00:13:27.57 >> The last time we visited the four schools in Manu'a, it as about 2 years ago, I will say.

00:13:33.24 >> Okay.

00:13:33.46 >> We were supposed to go last year, but then the COVID shutdown ...

00:13:37.23 >> Okay, yeah.

00:13:38.71 >> And we didn't have the chance to visit the schools.

00:13:42.30 >> Jessica, it sounded like COVID sort of impacted you all a little bit later maybe than ...

00:13:47.38 >> Right, right.

00:13:47.97 >> ... stateside, so you're even feeling the brunt of it even harder still than ...

00:13:54.79 >> Exactly.

00:13:55.22 >> Yeah.

00:13:55.79 >> I think when it hit our shores, it was completely dying out.

00:13:59.69 >> Yeah. So you're still ...

00:14:02.06 >> We didn't ...

00:14:02.36 >> ... in the throes of it.

00:14:03.85 >> Right. Got you.

00:14:04.23 >> Yeah. Well, tell me. What do you have coming up next? What plans do you have? What are you really kind of focusing on now that you hope will kind of move you even to the next level in terms of your data quality?

00:14:17.73 >> Actually, right now, we are looking forward to our DNS monitoring this year.

00:14:22.37 >> Mm-hmm.

00:14:22.89 >> So we are trying to ... Our leadership and our staff here at SPP, we're trying to get everything done before submission, which is in April.

00:14:32.53 >> April.

00:14:33.55 >> So that's the phase one.

00:14:34.66 >> Mmm.

00:14:35.01 >> Mm-hmm.

00:14:35.25 >> Mm-hmm.

00:14:35.86 >> So we are looking forward to this big, shocking visitation next year.

00:14:42.73 >> Mm-hmm.

00:14:43.21 >> But I believe we can manage, and I know I have a good team here, and we are working very hard as of right now to try to prepare for this decision next year.

00:14:55.49 >> Yeah. I think what it is now, Amy, is we're looking at ... American Samoa is now the go-to for the Pacific Islands ...

00:15:05.12 >> Mm-hmm.

00:15:05.46 >> ... because we are the first out of the Pacific Islands ...

00:15:07.79 >> Right.

00:15:07.88 >> ... to be visited.

00:15:09.79 >> Mm-hmm.

00:15:10.12 >> And we're placed in cohort two. The other islands or based in cohort three ...

00:15:13.72 >> Right.

00:15:13.81 >> ... and cohort seven, I think. And so now our hope is the big island is looking up to American Samoa to see how the visit goes and ...

00:15:21.47 >> Right.

00:15:21.75 >> ... what needs to be done and improved in their system stateside. But that's the biggest event for us ...

00:15:27.76 >> Mm-hmm.

00:15:28.00 >> ... is our visitation in September. They referred to it as ... We're a month late because we were supposed to be in October, but they moved it forward to September.

00:15:36.76 >> Mm-hmm.

00:15:37.33 >> Mm-hmm.

00:15:38.38 >> But that's not stopping American Samoa from doing what we need to do ...

00:15:41.32 >> Mm-hmm.

00:15:42.03 >> ... as a program, and ...

00:15:43.43 >> Mm-hmm.

00:15:44.76 >> You know, to be a data quality influencer, and the big event is to be able to grab all this data and go back to our leaders and say, "Hey, look at us. This is what we need. It's not only going to work for students with disabilities but for all people with disabilities on the island."

00:16:01.08 >> Mm-hmm.

00:16:01.86 >> There is a need. If I were to place it within what we have for OSEP in regards to, where do we place that? It's still at that, needs assistance, for American Samoa. It's because we don't have all the available resources ...

00:16:17.86 >> Resources.

00:16:18.75 >> ... for us here.

00:16:19.39 >> Mm-hmm.

00:16:19.91 >> So a lot of things are Amy, and her leadership team is ... Don't mind my jargon, but, you know ...

00:16:26.19 >> Yes.

00:16:26.30 >> ... gung-ho in trying to make that our students are provided for and the services are reaching our students that live in very difficult geographical areas on the island.

00:16:38.61 >> Yeah. If Internet access is an issue places, that's huge. We saw during COVID, distance learning and how much that came into play and how, if families and kids couldn't get online, that really has a big impact, and being able to enter the data is going to impact accuracy and all of that. So, yeah, you definitely have those things to deal with, too. But I think, yeah, it sounds like you're doing all you can, and I know we'll be in great shape when it comes time. And then you'll also ... You have your processes kind of documented, and you'll have your evidence, and all of that is great for you as an SCA to have in your hands moving forward, too, so you kind of get something out of it, I think, as well. And it really forces you to kind of think through your processes, right, and see maybe where there are gaps and where we need to make improvements and all of that. So I think it's a learning experience, for sure.

00:17:36.37 >> Yeah, during our meeting last month in Guam ...

00:17:40.44 >> Mm-hmm.

00:17:41.03 >> So what Jessica mentioned, all of the Pacific entities are all looking at us, and they're waiting for that moment for us to be done ...

00:17:51.73 >> Uh-huh.

00:17:52.81 >> ... get the feedback ...

00:17:54.11 >> Yeah.

00:17:54.30 >> ... of what was going on. And so we were like, "Yeah, just be patient. We are praying for ourselves, too, as well, to make a good thing out of it."

00:18:04.04 >> Mm-hmm.

00:18:04.43 >> So hopefully, yeah, we'll do good. But we'll share the information with them, as well. We like to help them out, too.

00:18:12.01 >> Right. Yeah, definitely. Maybe afterwards, you can come back on the podcast and talk about ...

00:18:17.73 >> Most of ...

00:18:18.11 >> ... talk about how it went. Yeah. And I'd like to kind of wrap up, asking if there's something you're really proud of, a success story or something you want to highlight related to your IDEA data.

00:18:31.79 >> I think we have a lot to be thankful for and to be proud of, especially for the special education program here in American Samoa. It's come a long way from stereotyping and the stigma from the community and the culture and traditions because, I guess, American Samoa, even though we're a territory, we're still culturally and traditionally rooted into our system. And so fast-forwarding now, we've been able to send people to get educated in regards to special education. We've been able to put teachers through school to get their BEDs, to get their master's to help provide services for students with disabilities here in American Samoa. But data-wise, we're able to be accountable for students with disabilities because then we can look at the numbers, and we can look at what services are provided, and it's here that we're looking at that there's a large turnover rate of teachers, and that's the one thing that we really need here on the island. But I'll let Dr. Boat elaborate more on the services for American Samoa and the big wow factor for us here.

00:19:43.94 >> Okay. Thank you, Jess. First, I would like to say, so thankful with our director and deputies with the big support from their end. I have a big support system here at the office with my leaders, my program directors, my staff and the best one here with our data manager because, without our data manager, we would never get the data from the schools and here at our office that we need to submit, all that. Also, I have very supportive teachers out in the schools that we always get the information, the data, from them as long as they get the information from the data person or the PDs. So they're good to go. But also, a big [foreign] and big thank-you to our TAs, Mr. Cesar, [Indistinct], Ms. Rhonda, Charlie. Yes, they've been so helpful with our being the best TAs for American Samoa. They've been so helpful, and they always make sure that we're on top of our roles and things that we need to ... They always knock on our doors, make sure that we get the e-mails that we need to submit right on time or at 2, 3 days before. So our data manager here is always connecting with them. Even us, too, we connected them to make sure that we need to make sure that everything is okay to go, and that's a big help from our TAs. And even with our DMS visitation next year, so they've been so helpful to us. Thank you so much, Amy.

00:21:16.11 >> Yeah. I also wanted to add in, Amy, with the data that's collected, and we're looking at our students, we begin to notice that over the previous years, most of our students were classified as specific learning disability. We're now moving forward. We've been able to diagnose students with disabilities under SLD. We've been able to diagnose students that have autism, emotional disturbance. We've had multiple disabilities, intellectual disabilities. So we're slowly getting the work done, and again, I would like to say thank you to our leadership team here. Without our AD, without the program directors, we wouldn't be able to push forward in our physical office, or Ms. Elaine Nico, to push forward in contracting our related services part ...

00:21:59.21 >> Mmm.

00:21:59.54 >> ... which is our speech and language and occupational therapy. And I know there were calls from stateside that most of the states are also experiencing this problem, too, that they're trying to find related service providers for students with disabilities, and we're experiencing the same thing maybe not as much but probably twice as hard as what's happening stateside. So we've been able to acquire services through telehealth for occupational therapy and speech and language. We also have Carlo.

00:22:30.21 >> Mm-hmm.

00:22:30.69 >> He's also provided assistive tech for us online, as well.

00:22:34.56 >> Mm-hmm.

00:22:34.88 >> So I'm proud of the fact that American Samoa is moving forward and providing. I'd also like to do a shout-out for Ms. Mari Shakazumal or Shakazuri ...

00:22:44.75 >> Yeah, Mari.

00:22:45.52 >> ... and Jennifer Wolfsheimer that's also part of our technical assistant ...

00:22:48.73 >> Right.

00:22:49.29 >> ... Chris Stacker and his crew ...

00:22:50.64 >> Mm-hmm.

00:22:51.44 >> Mm-hmm.

00:22:52.22 >> ... for always being there for us and the segment collaborations that we have for the online system, IDC, NASD ...

00:23:01.07 >> Mm-hmm.

00:23:01.77 >> ... everybody that's been awesome for American Samoa because we can't travel like they do stateside.

00:23:07.59 >> Mm-hmm.

00:23:07.89 >> You could just jump on a plane and pay a $200 ticket. For us, it's going to be almost $10,000 just to get off the island to go ...

00:23:15.04 >> Mm-hmm.

00:23:15.86 >> So we rely so much on the seminars and the webinars that we have being e-mailed out.

00:23:21.35 >> Mm-hmm.

00:23:21.82 >> And we make sure that we try to ... Even though it's 3 o'clock in the morning, we try to get online, so ...

00:23:27.42 >> Yeah, I always see you, Jessica, and I think, "Oh, my gosh. What time is it?"

00:23:32.28 >> So everything goes through our AD, and without our AD's approval, we won't be able to ...

00:23:37.67 >> Mm-hmm.

00:23:37.99 >> But a lot is moving forward for the program, and I am thankful for that and all our TAs and ...

00:23:42.69 >> Mm-hmm.

00:23:43.21 >> ... as such, yourself, as well ...

00:23:44.56 >> Mm-hmm.

00:23:44.78 >> ... for always being there for us.

00:23:47.99 >> Right.

00:23:49.66 >> Great. Well, you have some really amazing things happening, and I just can't wait to see what the next few months and few years bring as you keep progressing. And with EMS, I know we'll be great and love to have you back on and hear more about the good work you're doing.

00:24:07.71 >> Thank you.

00:24:08.22 >> So thank you both so much.

00:24:11.49 >> To access podcast resources, submit questions related to today's episode. Or, if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or, connect with us via the podcast page on the IDC website at IDEAdata.org.

View Details

"Data is everything and everywhere," and to prove it this installment of A Date with Data travels to the middle of the Pacific Ocean and the Northern Mariana Islands (CNMI). Join island-hopping host Amy Bitterman as she sits down with the CNMI public school system's Donna Flores and Nora Fujihira to talk about their journey to improve data quality; support students, teachers, and families; and work with their SEA to develop ever improving data governance policies. Pack your bags; you've got a date tonight.

View Details

Collaboration. Communication. Continuation. These are just a few keystones of improvement. In this second part of our deep dive into Pennsylvania’s SSIP journey, guests Dr. Laura Moran and Barbara Mozina share how they helped develop the ability of stakeholders—especially families and students—to provide feedback by creating and maintaining a continuous flow of information and by hosting focus groups dedicated to creating resources to boost graduation rates. It’s the culmination of our double date in the Keystone State, so don’t miss out.

View Details

Stop us if you’ve heard this one: A state wishes to increase graduation rates and decrease drop-out rates. That’s what happened in Pennsylvania, where Dr. Laura Moran and Barbara Mozina applied lessons learned to the second iteration of their state’s SSIP. In the first episode of this two-part A Date with Data, host Amy Bitterman hears all about it, including their efforts to move districts beyond being “data rich and information poor,” developing an online “early warning” data platform, and protocols for creating and sustaining data teams. You’ll be the richer, too, for tuning in.

View Details

At some point, every one of us has been new at something. A hobby. A job. A relationship. On this episode of A Date with Data, host Amy Bitterman talks about first steps and initial impressions with the North Dakota Department of Public Instruction’s Alyssa Kramer and Part B data management coordinator with the West Virginia Department of Education, Traci Tuttle. From internal data collection and reporting to strategies for other new data managers, together Alyssa and Traci discuss tips, resources, and EDFacts file specifications, all from the newbie perspective. Whether you’re a rookie listener or seasoned veteran, this is one play-by-play you won’t want to miss.

View Details

“Improvement begins with data.” These are words to live by and a philosophy held in common by the Massachusetts Department of Elementary and Secondary Education’s Brian Coonley and the Kentucky Department of Education’s Jarrod Slone.

In this episode of A Date with Data, host Amy Bitterman gets to the bottom of these two commonwealths' not-so-common strategies to support districts, which include a focus on increased transparency and data stewardship. Jarrod and Brian also discuss how root-cause analyses and long-term strategic planning can lead to meaningful, substantive improvement.

View Details

When it comes to high-quality data, there are some questions we all ask ourselves. What does it mean to be a data-quality influencer? Am I a data-quality influencer? And, if not, what must I do to become one?

To unearth the answers, our latest episode of A Date with Data visits the Gem State, where host Amy Bitterman discusses this multi-faceted issue with the Idaho Department of Education’s Alisa Fewkes and Debi Smith. They talk about polishing up a school’s and district’s ability to read and analyze data through drill-down trainings that the department offers around the state. Can you see yourself reflected in their strategies of creating strong partnerships between data staff and program staff? Or of making certain that general education staff is part of important team-making decisions? What about holding meaningful reviews to discuss enhancements going forward? Join us to find out. This one’s invaluable.

View Details

On the newest episode of A Date with Data, we “Wander Indiana” as Host Amy Bitterman sits down with Brandon Myers, Part B Data Manager with the Indiana Department of Education. Listen in as they discuss Brandon’s work as a data-quality influencer: collecting, organizing, monitoring, and communicating information about data. Brandon also shares how he ensures that schools and stakeholders understand how the data they are required to report get used. Join us in the Hoosier state, won’t you?

View Details

Welcome to Nashville, Tennessee, home of country music, hot chicken, and southern hospitality. In this episode of A Date with Data, we’re on stage at the 2022 IDC Interactive Institute in Music City, USA. Join us as host Amy Bitterman performs a duet with Amy Patterson, IDEA Part B Data Manager with the Kentucky Department of Education, to learn more about her role as a data-quality influencer. Up next, Amy two-steps with Richelle Davis from the U.S. Department of Education’s Office of Special Education Programs for her perspective on the importance of states and districts understanding the “why” behind collecting high-quality data—including how the data drive policy and decisionmaking. Be sure to join us, won’t you? This one’s going Platinum®.

View Details

It's the age-old question: What’s the secret to a long-lasting, meaningful relationship? For Episode 5 of A Date with Data, we travel to the first state to find out. For 20 years, the Delaware Department of Education has partnered with the state’s Parent Information Center to better facilitate family engagement. How do these partners maintain their strong bond? Host Amy Bitterman sits down with Mary Ann Mieczkowski, Barbara Mazza, and Meedra Surratte to talk about collaboration, specific programs, and the importance of creating spaces where family voices can be heard. You don’t want to miss this engaging conversation.

View Details

Wisconsin is striving to embed IDEA data culture throughout the state’s Department of Public Instruction. Learn how Wisconsin is working to achieve a strong, shared data culture through initiatives such as collaborating with Title I and Every Student Succeeds Act staff to create joint federal notification reports and creating dashboards for districts to drill down into their data.

View Details

Amy chats with Ginger about building the capacity of districts and other stakeholders in Oklahoma to understand the changes to the SPP/APR and stakeholder engagement requirements. Ginger shares key strategies for engaging stakeholders in target setting.

View Details

In this episode, we throw a spotlight on the implications of both the pandemic and the changes to the SPP/APR for FFY 2020—2025 for Arkansas’s data quality, results, and trends. Jody and Amy discuss how Arkansas mitigated the data challenges it encountered due to the pandemic and SPP/APR changes. Jody also shares the state’s strategy for successfully communicating the changes to the data and the SPP/APR indicators to districts and other stakeholders.

View Details

During our first episode, we explore how the pandemic affected Georgia’s Part B 618 and SPP/APR data trends and quality. The good news? It’s far from doom and gloom. Dawn and Linda highlight many bright spots, including how they have harnessed virtual technology to improve collaboration, connectivity, and stakeholder engagement. They also focus on the positive effects the pandemic has had on Georgia’s data culture.

View Details

Welcome to 'A Date with Data'... Special education staff have been asking to hear from staff in other states and districts about what they’re doing that’s working and lessons they’ve learned, so we’re throwing a spotlight on your IDEA data stories. Join your host, Amy Bitterman, as she chats with state and district special education staff, who, just like you, are dealing with IDEA data every day.

A Date with Data: Recent Episodes

IDEA Data Center (IDC)

Episode Transcript ###