AD: The opening slide is headed "OSEP TA&D Program Performance Measurement," subheaded "GPRA Data Collection Involving 84.326 Centers, March 19th, 2018," and displays the U.S. Department of Education logo. Tina: Alright, I think I'll just go ahead and get started. This is Tina Diamond from OSEP, and I am the Program area lead for the TA&D program since last year. For those of you joining today, if you weren't on the call last year, welcome. I'm looking forward to working with you on the GPRA process this year. I also wanted to welcome members from our CIPP GPRA contactor partners here in this process. Elaine Carlson is with us from Westat, Kerri Wills and Myriell McKinnon. Elaine, did you guys want to introduce yourselves? Elaine: Sure, hi everyone. I see some familiar names on the list, so it's nice to be working with those of you again who I have already met. And for the new folks, I look forward to getting to know you. Tina: Thanks. As is typical, we don't have a lot of control over machines in our office computer, and mine was pushed to restart at about 10 minutes before the hour so I do have my slides but I'm not able to see what everybody else is seeing here on the WebEx. But, I'm going to do my best to improvise. I also just wanted to get a sense of who we have with us here on the phone line today. I see that a good number of people have already muted themselves but it'd be nice to hear which projects are represented, and I also know that we're being joined by some of the OSEP Project Officers. So, if we could first just hear from who is on the line from a TA Center. Just tell us who you are and what center you're with, that would be great. Michelle Rovins: Michelle, NCIS. Tina: Hi Michelle. Melanie: Melanie Reese from CADRE Okay Michelle from NCIS and Melanie from CADRE Tina: There is somebody talking, but I couldn't hear you. I think you might be too far away from the speaker. Okay, I did hear you this time Katherine, but just barely, so if you have any questions, your connection doesn't seem super strong. It might be best to put it in the chat. Katherine: That's what I'll do, thanks! Tina: Great. I also see that Kent McIntosh from the PBIS Center checked in there in the chat pod. We have several projects represented here. And then we have a good number of OSEP staff on the line as well. I see Jennifer Coffey, JoAnn McCann, Louise Tripoli, Perry Williams, Polly Maccini and hopefully I didn't miss any of our OSEP Project Officers. Thank you for all joining us today. We're going to go ahead and get started with today's webinar. AD: The slide, titled Webinar Objectives, displays the following points: Enhance awareness of the Program GPRA Performance Measures requirements; Provide overview of procedures related to OSEP TA&D Program GPRA performance data, specifically, Collection and information gathering, Analysis and Reporting; and Respond to grantee questions. Tina: So the purpose of bringing everybody together is to really talk about the GPRA process. To explain first off, what the process is and to provide you with an explanation of the procedures that you'll follow as a project that is participating in this evaluation. And then most important, really, is to answer any or all questions that you have related to the GPRA process so that you can participate as best as possible. So we'll go on to slide 3 and just to give you a little bit of background, the Government Performance and Results Act of 1993 first required all federal agencies to establish and report annually on goals, measures, and targets, and that law was reauthorized in 2010 as the "GPRA Modernization act of 2010." And so GPRA is required to conduct performance assessments of all government programs for the purpose of assessing agency performance and improvement. OSEP works in partnership with our office of Budget Service to determine whether or not we're meeting our performance measures. Next slide. So OSEP does report the progress annually to OMB as well as Congress. You'll see information used from this GPRA process in the Congressional justification for any proposed projects that we would like to see in upcoming budgets. The information that's passed on is an aggregate picture of the performance of our Technical Assistance and Dissemination program. The review of our performance affects whether or not we get continued funding of our IDEA programs. OSEP does use the information, looks at it annually to make improvements within the TA&D program. And we also report the results out to our TA&D projects. The results of this last year's analysis will be presented at this summer's Project Directors' Conference. The purpose of the program is to improve services and results for children with disabilities, to promote academic achievement, and to improve results for children with disabilities by providing TA and supporting model demonstration projects, disseminating useful information, and implementing activities that are supported by scientifically based research. So just for clarification purposes, today's webinar is focused on our TA&D Centers. We'll be conducting a separate session tomorrow that will be focused on our OSEP funded Deaf/Blind state projects. Next slide. We have two different types of measures that are included in that TA&D program. We have annual performance measures as well as long-term performance measures. We'll be focusing today on the annual performance measures and the procedures that will be used to collect information from each of the projects participating. So annually, we look at the quality of new services or new products. We look at the relevance of new services or new products, usefulness of new services or new products, and the efficiency of our TA&D program as a whole. This measure is looking at milestones and the cost associated with meeting those milestones. And then the long-term measure is data that we collect typically every two years and looking at whether or not we're seeing implementation of evidence-based practices that are supported by our TA&D Centers and states or local districts, and whether or not our effective models are being promoted within states and districts. As I said, we won't be collecting on this one this year but I do like to give you the full picture of what the GPRA process looks like for us. So today we'll be focusing on how the quality, relevance, and usefulness annual measure as well as the efficiency measure is collected. So how is quality rated by the panel? Well, first off we should say, before I dig a little too deep, that we have two different panels that will be assessing the information that the TA Centers submit to the CIPP project as part of the GPRA process. There's a science panel and a stakeholder panel. Depending on how you indicate your products or services will depend on which panel looks at your information that is submitted. The quality is the percentage of special education TA&D products and services deemed to be of high quality by an independent (science or stakeholder) review panel. The panels look at substance. Does the product content or the content delivered through the service reflect evidence of conceptual soundness and quality, grounded in recent scientific evidence, legislation, policy, or accepted professional practice? And then the panel also looks at communication. Is the product content or the content delivered through the service presented in such a way so as to be clearly understood, as evidenced by being well-organized, free of editorial errors and appropriately formatted? Moving on to the next slide, "How is Relevance rated by the Panel?" The percentage of special education technical assistance and dissemination products and services deemed to be of high relevance by an independent stakeholder panel. So in the case of relevance, all products and services that are submitted would go to the stakeholder panel, whereas under the quality, if you indicate that you have a scientific product, that would go and be reviewed by the science panel for evidence based products. So the relevance, for the rating there, the panel will look at the need, the pertinence, and the reach. Under need, does the product content or the content delivered through the service attempt to solve an important problem or deal with a critical issue? Under pertinence, does the product content or the content delivered through the service address a problem or issue recognized as important by the target audience(s)? Under reach, to what extent is the product content or the content delivered through the service applicable to diverse segments of the target audience(s)? AD: The introductory text under the title "How is Usefulness Rated by the Panel?" reads: Percentage of Special Education Technical Assistance and Dissemination (TA&D) Products and services deemed to be useful by an independent (Stakeholder) review panel. Then on the next slide, for usefulness, the stakeholder panel will look at ease and suitability. Does the product content or the content delivered through the service address a problem or issue in an easily understood way, with directions or guidance regarding how the content can be used to address the problem or issue? And for suitability, does the product or service provide the target audience(s) with information or resources that can be used again or in different ways to address the problem or issue? And then the final annual measure that we will be looking at is efficiency. And that's the cost efficiency of the technical assistance and dissemination program. And we look at the percentage of milestones achieved in the current APR period and the percentage of funds spent during the current fiscal year. So in your annual performance report you'll submit, you will have submitted in the previous year a list of milestones that you plan to achieve. We'll then look at whether or not those were reported as being achieved in your APR. And we'll consider the draw down in our G5 system, so how much fewer funds have been drawn down. On the next slide, who participates? All of our 326 or TA&D Centers who are in their second to fifth year of funding in FY2017 are part of the pool for the GPRA measures as well as any centers that are operating under a no cost extension during FY2017. We should note that Centers that are operating in their first year of funding during FY2017 are not included in the data collection. Let's talk a little bit about the data collection for annual measures. And again, those are the quality, relevance, and usefulness measures. Each project will be asked to generate a list of the new products and services developed during FY2017. You'll submit that list to CIPP, and CIPP will randomly select one product and one service from the list. Then CIPP will provide you with a description template, and you'll provide a description, a very detailed description, of the selected new product and new service. The descriptions and the products are sent to a science expert panel (if they're considered to be an evidence based product or service) to rate the quality and to a stakeholder panel who will rate the quality if it's a policy-related product or service. And the stakeholder panel also looks at and makes the assessments under relevance and usefulness. On the next slide, we talk about the data collections for products, or what is a product. A product is a piece of work (in text or electronic form) that informs a specific audience on a topic relevant to outcomes for infants, toddlers, children, and youth with disabilities. The examples of products can include journal or informational articles (including web-based instructional materials), booklets, pamphlets, manuals, DVDs or CDs, multimedia kits or modules, and even PowerPoint presentations. What are not considered products are things such as databases or project websites, meeting agendas or materials developed for one time use by an individual. Conference proceedings are not considered as products nor are contact lists or evaluation reports. On the next slide, a service is defined as work that provides information to a specific audience relevant to outcomes for infants, toddlers, children, and youth with disabilities. OSEP recognizes three levels of TA services: General/Universal technical assistance, Targeted/Specific technical assistance, as well as Intensive/Sustained technical assistance. For the purposes of the performance measure review, again, maintaining a project website is not considered to be a product or a service. On the next slide we have some examples of services: conducting training or technical assistance; providing captioning, video description, Braille, or other accessible formatting of text or media; leading and convening informational meetings; and responding to inquiries from a targeted population. Again, these are just examples, and there are many other things that would fall under services. So on the next slide, we have information about the new product or service description guide. Again, once you're notified which of your products and services are selected, you'll complete a service description or a product description guide. They're important to the GPRA review process, and we really ask that you put an expansive amount of your time and attention into submitting very descriptive information in your submission. This is what the review panelists will look at. So we really need a complete, detailed, and clearly written description to make it easier for the panelist to rate your product or service accurately. There's going to be some consulting tips that will be sent out, and you can look at those to complete the description guide. It will be provided by CIPP. So, on the next slide data collection for annual measures, Elaine is going to talk in more detail about the submission process and the system that they've created. This will be new for OSEP this year, having this process in place. She'll provide the critical information on that. You'll be able to upload an electronic copy either in Word or PDF of your completed guide and any supporting materials to cippsite.org. Be sure that your URL addresses or any supporting materials for your products and services are marked clearly. So that is if you want the panelists to look beyond what's in the description, that information needs to be prominent. If you have questions along the way, you may email CIPP-GPRA@westat.com and they'll be responsive to any questions about the submission process. And then the deadline for submitting any information is April 23rd. And then on the next slide, I'm just going to talk a little bit about the long-term measure and again this is one that we will not be collecting in 2018. But for the long-term measure, we look at the percentage of states receiving TA&D program services regarding scientifically- or evidence-based practices for infants, toddlers, children, and youth that successfully promote the implementation of those practices in their school districts or service agencies. How do we do that? On the next slide, data collection for the long-term measure, we select centers who self-report that they worked intensively with states to implement evidence-based practices. Those centers then provide a list of states with which they have worked to promote evidence-based practices to districts and service agencies. And then CIPP randomly selects one state on which to focus data collection and contacts center staff to learn more about the state and local EBP implementation. So centers are asked, again, if they've worked intensively in states during the most recent five fiscal years and if they have, they are selected to be a part of that long-term measure. Alright, and then on the next slide I'll go over the schedule and then we'll turn it over to Elaine. So, the TA centers who are participating in the GPRA process were notified the week of March 5th. You'll be responding with materials, your list of services and products by March 26th. And then CIPP will notify your center which product and which service was selected the week of April 2nd. Then they'll be making that request for the detailed product or service description that will be due by April 23. Actually, I'll pause there and see if there's any questions and then we'll shift over to Elaine so you learn more about specific information for how to submit in their system. If you have a question, feel free to either put it in the chat pod or to unmute yourself. I'm not seeing anything in the chat pod and I'm not hearing any questions. So Elaine, I'll go ahead and turn it over to you. Elaine: Okay, thank you Tina. We're using a slightly different process this year for collecting materials for the QRU Review. We're hoping that it will be easier for you, easier for the panelists who will be reviewing your materials and, of course easier for OSEP as well. So, we have set up a website that Tina mentioned previously, cippsite.org, where you'll be asked to upload both your list of services and products and then your description guide once your service and products have been selected if you had a chance to do that. There is a quick start guide provided as well, so much of this material will be provided in writing as well as here today on the slide. So if you don't catch everything don't panic. When you go to cippsite.org, and you go to login, you'll see a login page like this. You'll be asked to enter your email address and everyone has a temporary password, that's Password, with a capital "P"pound 1-2-3-4-5. Once you login the first time, you'll be asked to reset your password. So use the password guidance that's given about what the requirements are for a password; they're not too horrible. AD: Slide depicts the CIPP Homepage, showing a menu on the left that includes Home, General TA, GPRA Reporting and Password Reset; the title "Center to Improve Program and Project Performance"; three links entitled My Actions, with an icon above each, which include "Access Resource Library," "Submit TA Request" and "Conduct GPRA Reporting." "Conduct GPRA Reporting is circled on the slide. Below the Actions is the title My Projects, with text that indicates the list is empty. To the right of the Actions is a column entitled Announcements, with text that indicates there are no items to view of the Announcements list. Elaine: Ok, once you're in, you'll be taken to the home page. Just fair warning before I go into more detail, if you're not interacting with the page for 30 minute you'll time out. So just be aware that you can let it site for a little while, but not too long or you'll lose what you haven't saved. AD: Slide depicts the "Conduct GPRA Reporting" page of the CIPP website; the title "Conduct GPRA Reporting" and descriptive text below. The QRU Review link is circled. Elaine: So once you get to this page, if you click "Conduct GPRA reporting" it will take you to the "QRU Review" page, and you'll select that. AD: Slide depicts the "Conduct GPRA Reporting" page on the CIPP website; and the title QRU Review. The TA&D Panel link is circled. Elaine: It'll take you to "TA&D Panel" and that will take you to "TA&D Panel Documents." AD: Slide depicts the "Conduct GPRA Reporting" page on the CIPP website; and the title TA&D. The TA&D Panel Documents link is circled. Elaine: If it seems as though it's a bit redundant, it's because other people will be seeing more than you will or different things than you will, I should say. So the reviewers, for example, will see slightly different materials than you will. Likewise, the OSEP staff see the QRU review for all of the program areas and not just for TA&D. So if some of these pages look fairly blank, that's why. AD: Slide depicts the "QRU Review TA&D Panel" page on the CIPP website; and instructional text on how to share files. Controls for New documents, Upload, Sync, Share a More tool, an All Documents link and a search bar are above two folders for Grantee Documents and Guidance Documents. Instructional text at the bottom reads "Drag files here to upload." Elaine: After you go into the "TA&D Panel Documents" section, you'll see that there are two folders: one called "Grantee Documents" and one called "Guidance Documents." The "Grantee Documents" folder contains the subfolder for each of your grants. I'll show that in a minute. And then the "Guidance Documents" folder has more general information pertaining to the QRU review. So for example, it will have a copy of the form that the panelists are going to use to review your product and service. So if you're interested in seeing what the criteria are that they're using, you'll be able to see it there. AD: The TA&D page on the CIPP website has expanded downward to show many file folders below the file tools. Folder names that precede with "DB" are indicated to mean "Deaf-Blind Projects"; Folder names that precede with "TA" are indicated to mean "TA Projects". Elaine: So once you go into "Grantee Documents," you'll see this page, and it lists again for each of the Deaf/Blind projects and each of the T/A Center projects, it gives your grant number. The DB ones, all the Deaf/Blind ones start with DB, and all of the other ones start with TA. So if you don't find your folder, send us a quick email and we'll fix whatever problem we might have had. But, we think everybody should have a folder there now. AD: Slide depicts the "TA&D Panel Grantee Documents" page on the CIPP website; and instructional text on how to share files. Controls for New documents, Upload, Sync, Share a More tool, an All Documents link and a search bar are above a file icon with the title "Name" and a check-mark beside it. Instructional text at the bottom reads "Drag files here to upload." The upload link is highlighted. Elaine: So we are asking that you upload your materials to the website. Again, both the list of product and services that are new this year as well as the product description guide and the service description guide that you complete for the one product and the one service that we select. Again, it's not too difficult. Within the "Grantee Documents" page, you'll be under your own grant folder and then you'll just click this upload link and search for the document on your local machine to upload to the site. AD: Slide shows the Add a Document screen: The screen shows an upload window entitled Choose a file, a link below that reads "Upload files using Windows Explorer instead," and a checked box next to the legend "Add as a new version to existing files; a folder window with a Choose Folder button entitled Destination Folder; and a text box entitled Version comments. Okay and a Cancel buttons are at the bottom. Elaine: If you have, this is what it looks like when you browse your computer. I'm sure everyone's familiar with that. If you have any problems at all, please feel free to email us. The email that Tina mentioned earlier, CIPP-GPRA@westat.com is set aside exclusively for this process. We'll be monitoring that email address and we'll get back to you with whatever information you require. So we'll be happy to take any questions either about Tina's portion of the presentation or about this. Michelle is asking whether we'll share the PowerPoint after the webinar. I believe the plan is to put it on IDEAs that Work, is that right, Tina? Tina: Yes, that's right. But if anybody needs it more immediately, I can email it to you. Elaine: It will take a little while to get it on there. Is there anybody who has not received the initial email from CIPP notifying them about the GPRA process and their participation? Okay, guess that's a good sign. And my other question I guess is whether anybody has already tried logging onto the website?... We'll take that as a no. If you don't mind, it's helpful for us if, after we hang up, if you can just go on and make sure you're able to login. That would be helpful because, of course, we never if there might be a glitch until, until people try it. Again, if for example we have the wrong email address for you, your login might not work. So that's my biggest concern is that we have an incorrect email addresses or that we again don't have your grant number listed. Those would be the two things that I'd be most concerned about. So if people would do that after the webinar, I would really appreciate it. Other questions, comments, concerns? Tina: You guys are a quiet group today. Elaine: Katherine, I see your comments and I'll make sure that your email is included in everything instead of just going to Tina. Tina: And this is Tina, so if anybody needs the slides emailed to them directly, if you could just send me a quick email at tina.diamond@ed.gov and I'll send it out to you. I'm hearing no additional questions or seeing no additional questions. I will go ahead and wrap this up early for today. We had allowed one hour thinking there might be some more questions especially in regards to the new system but Elaine, I think you explained it very clearly. I appreciate all the hard work that has gone into developing the system because it certainly will streamline this process. So thank you to you and your whole team over there, I think it looks really good. Elaine: You're very welcome. Tina: Great, well again if there are any follow up questions or as soon as we hang up the line you think of something you wanted to ask that you didn't want to ask online, feel free to reach out to either me or to the dedicated CIPP-GPRA email address, and we'll be responsive to your question. Thank again for taking time out of your schedule, all of the TA center folks as well as OSEP colleagues here. I appreciate your time and thank you for participating in this important program.