Background
Data validation is a series of internal controls and quality assurance methods established to verify the accuracy, validity, and reliability of data. This framework requires consistent approach across programs to ensure that all program data consistently and accurately reflects the performance of each grant recipient.
The purpose of data validation is to:
- Verify that the performance data reported by the State of Minnesota, the Local Workforce Development Areas (LWDAs), and other local partners are valid, accurate, reliable, and comparable across programs.
- Identify anomalies in the data and resolve issues that may cause inaccurate reporting,
- Outline source documentation required for common data elements, and,
- Improve program performance accountability through the results of the data validation efforts.
DEED is required to report Minnesota's performance outcomes to the DOL using the Participant Individual Record Layout (PIRL) file. The PIRL file provides a standardized set of data elements, definitions, and reporting instructions that is used to describe the characteristics, activities, and outcomes of WIOA program participants.
Policy
Data validation must be conducted to ensure, to the maximum extent feasible, the accuracy of the data entered by WIOA core and non-core workforce development programs into the state's Management Information Systems (MIS), which are currently WorkforceOne (WF1) and MinnesotaWorks.net (MNW). Data element validation is required to be conducted on an annual basis (at a minimum) to maintain and demonstrate system integrity, assess the accuracy of participant data in the MIS, assess the accuracy of the PIRL files, and identify and correct problems associated with data entry or PIRL file reporting. The source documentation for required data elements can be found in the attachment section of this policy.
Core WIOA and non-core programs subject to data validation, include:
- WIOA Title I (Adult, Dislocated Worker, & Youth),
- WIOA Title III Wagner-Peyser Employment Services (including Migrant and Seasonal Farmworker services),
- Trade Adjustment Assistance (TAA),
- National Dislocated Worker Grants (DWG).
Senior Community Service Employment Program (SCSEP) data validation will continue to be conducted per DOL guidance and requirements outside this data validation policy.
The State of Minnesota responsibilities:
- Maintain written procedures for conducting data validation, including describing processes for identifying and correcting errors or missing data;
- Provide annual data validation training for DEED and LWDA staff;
- Provide technical assistance to local areas;
- Maintain monitoring protocols that ensure program staff are following the written data validation policy and procedures and take appropriate corrective action if those procedure are not being followed;
- Conduct an annual review of program data for errors, missing data, out-of-range values and other data anomalies;
- Document when missing and/or erroneous data is corrected; and
- Review the effectiveness of the data validation process and policy as needed.
Local Workforce Development Areas (LWDAs) responsibilities:
- Develop internal controls to ensure data reported in state MIS (i.e. WF1 and MNW) is valid, reliable, and complies with source documentation identified in this policy;
- Train subrecipient staff on the importance of correct data entry and allowable source documentation on an annual basis, at a minimum;
- Review program data for errors, missing data, another anomalies, including missing documentation, on a quarterly basis(recommended);
- Ensure appropriate staff receive annual data validation training;
- Maintain WIOA participant records and source documentation per state and federal records retention rules; and
- Ensure DEED staff have access to source documents by uploading participant documents into the Electronic Document Storage (EDS)section of state MIS (currently only available in WF1).
Frequency of Data Validation
For PY 2023, DEED will conduct data validation of WIOA Adult, WIOA Dislocated Worker, WIOA Youth, Wagner-Peyser, National Dislocated Worker, and TAA participant files once during the program year.
For PY 2024, DEED will conduct data validation of WIOA Adult, WIOA Dislocated Worker, WIOA Youth, Wagner-Peyser, National Dislocated Worker, and TAA participant files once during the program year for each LWDA.
Following the annual assessment of the effectiveness of the data validation tool and procedures, Minnesota will consider the feasibility of increasing the frequency and scope of data validation in PY 2025.
Data Validators
Data Validators are Department of Employment and Economic Development (DEED) staff who are independent of the data collection process. CareerForce and local area staff will not validate their own data.
Data Validation Procedures
DEED's data validation procedure includes: 1) Stratified Random Sample of Participant Files; 2) Notification& Document Request; 3) Source Documentation Review; 4) PIRL Data Element Review; 5) Recording Data Validation Results; 6) Correction of Data Errors and 7) Final Report
1. Stratified Random Sample of Participant Files
DEED developed a sampling tool which guides the state in determining the number of records included in data validation. The tool includes WIOA participant counts, by LWDA, from each program reported in the PIRL. The WIOA participant population in the sampling tool includes each distinct cohort included in WIOA Annual Performance Reports: served participants, exited participants, second quarter after exit employment metric and median earnings participants, fourth quarter after exit employment metric participants, credential attainment metric participants, and measurable skills gain participants.
2. Notification & Document Request
The DEED data validator will notify the local area 10 business days prior to the beginning of the review period. The local area must upload required participant source documentation not already uploaded into Electronic Document Storage (EDS) so the data validator can conduct the review. Co-enrolled participants may be selected for data validation; therefore, it is important to ensure the required source documentation is uploaded into EDS. See DEED's EDS Policy for more details.
3. Source Documentation Review
The DEED data validator compares the participant data from the MIS against the supporting source documentation in EDS to assess the accuracy of participant records and to ensure compliance with federal definitions. The source documentation for required data elements can be found in the attachment section of this policy.
4. PIRL Data Element Review
The data validator compares the data elements from the PIRL file to the values recorded in the MIS to assess the accuracy of the PIRL file. The method for mapping MIS data fields and values to PIRL data elements and value scan be found in DEEDs data validation tool.
5. Record Results
The results of data validation for each PIRL data element for each participant file reviewed will be recorded within DEED's data validation tool. PIRL data element values that correctly correspond with MIS data values and that are supported with allowable source documentation will be marked, 'Y'. PIRL data elements that do not correctly correspond with MIS data, OR that are not supported with allowable source documentation, will be marked, 'N'. Not applicable ('NA') will be marked for data elements that do not apply to the participant file. DEED data validators will also record notes within the tool to document the reason why a PIRL data element can't be validated.
6. Correction of Errors
Each local area will securely receive the data validation results for all participant files reviewed, which will include the audit worksheets for each PIRL data element. Local areas will have30 calendar days to correct any data entry or source documentation errors and return their corrections to DEED data validators. When source documentation is missing, local areas may also use this time frame to provide such documentation.
DEED WIOA program administrators will receive the data validation results for all participant files reviewed, which will include the audit worksheets for each PIRL data element. WIOA program administrators will have 30 calendar days to correct any mapping errors between the MIS and PIRL file and return their corrections to DEED data validators.
7. Final Report
At the conclusion of the data validation review, a final report will be emailed to each local area and DEED WIOA program administrator. This report will contain a summary from the data validation review, including:
a) The number of files reviewed, and
b) A summary of error rates by data element.
Records Retention
All records associated with the data validation review will be stored electronically in accordance with relevant state and federal records retentions requirements. See Attachment V for the retention schedules of participant information and uploaded source documents saved within WF1 and MNW.
The state will be responsible for retaining information reviewed and used at the state level, completed Data Validation Tools, email correspondence, Data Validation reports, training documents, and any other information pertinent to Data Validation.
This storage period does not invalidate any other requirement to store records for a longer period.
Annual Evaluation
An annual evaluation of DEED's data validation policy and procedures will be conducted within 135 days of the end of the program year. Revisions of the data validation policy and procedures, including the data validation tool and sampling approach, will be considered based on error rates, trends in common data accuracy issues, and identification of issues by data validators and program administrators. Any policy or procedure changes will be included in annual data validation trainings.
Monitoring Protocols
DEED monitoring guides will include data validation questions, such as the steps LWDA regularly take to review program data for errors, missing data, and missing documentation. Program monitors will also observe whether local partner staff are following written data validation and source documentation procedures. If data validation procedures are not being followed by the local area, corrective actions will be issued within the monitoring report and follow-up will be required by the partner to ensure corrective actions are satisfied.
Annual Data Validation Training
All case managers and local area staff whose duties involve data entry or data validation will be provided an annual training. Topics will include data integrity, purpose, and procedures involved with data validation. This ensures all individuals who contribute to either data entry or data validation understand their roles in ensuring valid and reliable data.
Additionally, training will be adjusted yearly to incorporate an overview of data validation results and any corrective actions during the past year, identification of any trends in data validity, and any updates to the policy or procedures.
Definitions
Case Notes - Case notes refer to electronic notes entered in Workforce One (WF1) by the case manager that identify at a minimum, the following: (a) a participant's status for a specific data element, (b) the date on which the information was obtained,and (c) the name of the case manager who obtained the information.
Confidence Interval - A number between zero and100 representing the likelihood a result can be reproduced using a new sample. For example, a confidence interval of 70 would mean that, if another 100samples were used, 70 of them would produce the same result. As the confidence interval decreases, the odds that the sample is representative of the total population decreases.
Cross-Match- A cross match requires validators to find detailed supporting evidence for the data element in a database. An indicator or presence of a Social Security Number in an administrative non-WIOA database, i.e., a database not maintained by a WIOA core program such as data from the State's Department of Motor Vehicles, is not sufficient evidence for a cross match. State validators must also confirm supporting information such as dates of participation and services rendered. States must have data sharing agreements in place as appropriate.
Data Validation - A series of internal controls or quality assurance techniques established to verify the accuracy, validity, and reliability of data.
Data Validators – DEED staff who are responsible for completing the data validation of participant files.
Data Element Validation (DEV) - The federally mandated process by which the state annually assesses the accuracy of prescribed data elements in randomly sampled participant files against source documents in program files for compliance with federal definitions (refer to TEGL 07-18).
Electronic Records- Electronic records are participant records created, stored, or transferred in a form that only a computer can process and are maintained in the State's Workforce One(WF1) or MinnesotaWorks.net (MNW)management information systems. Electronic records as sources of documentation should be interpreted as information entered into Workforce One(WF1) or MinnesotaWorks.net (MNW) (e.g. case notes and/or uploaded documents)and can be numeric, graphic, or text.
Electronic/Digital Signatures- Electronic/digital signatures from the participant may include an email, text, or unique online survey response may be considered an electronic signature or verification; but it must be participant generated and traceable to the participant.
Margin of Error – A number between zero and 100 representing the likelihood a result from a sample would match the result for the entire population (accuracy). For example, surveys provide a percentage-point estimate, above or below their sample results, where they believe the true population result lies. As the margin of error increases, the confidence in the accuracy of the results decreases.
Participant Individual Record Layout (PIRL)- The PIRL is a data file with a standardized set of data elements, definitions, and reporting instructions that is used to report on the characteristics, activities, and outcomes of WIOA participants. DEED is required to report Minnesota's performance outcomes to DOL using the PIRL file.
Performance and Technical Management (PTM )Staff – DEED staff who manage many of DEEDs products and provide analysis and research in support of data-based decision making at the agency. PTM staff coordinate the creation and submission of the PIRL files and design and maintain the data validation and sampling tools.
Self-Attestation - Self-attestation is a written or electronic/digital declaration of information for a particular data element, signed and dated by the participant. Electronic/digital signatures from the participant may include an email, text, or unique online survey response may be considered an electronic signature or verification; but it must be participant generated and traceable to the participant. Local partners must retain and upload documentation of the self-attestation into Electronic Document Storage (EDS) of Workforce One(WF1).
Stratified Random Sampling - A method of sampling that involves the division of a population into smaller subgroups known as strata (common examples include gender or race). Before sampling, the percentage of the population in each strata is calculated. When the sample is pulled, the percentage of each strata in the sample is matched to the percentage of the strata in the population. This ensures the sample accounts for differences in results that may be caused by differences in experience between strata.