Evaluating Impact

Overview of Evaluation

When implementing a new career pathways initiative, it is important to embed evaluation activities into the design. Determining success criteria should be part of the planning process. This includes identifying targets, setting goals, and defining metrics. By creating tracking mechanisms and data collection procedures, you will be able to continuously improve the quality of your program and demonstrate progress toward goals and intended outcomes.

Nebraska and Virginia conducted formative or process evaluations to assess progress toward implementation of the project design and changes over time, and track the use of resources and progress toward objectives. Process measures and evaluation data collection procedures were designed to improve performance through feedback from periodic assessment.

Key features of the evaluation should include:

• Document model implementation
• Track challenges and solutions
• Collect feedback from participants
• Measure outcomes

The second prong of the model demonstration evaluations focused on summative assessment or outcome measures. The primary objective was to report on the model demonstration’s progress toward achieving the targeted outcomes, specifically as defined as part of the Government Performance and Reporting Act (GPRA) measures defined in RSA’s Notice Inviting Applications.

Targeted Outcomes

It is important to create program objectives and identify specific outcome goals when implementing a new program. The Career Pathways Demonstration projects identified three target outcomes: 1) Increase number of distinct career pathways accessed or created, 2) Increase number of individuals who achieve competitive integrated employment, and 3) Increase average weekly wage and employer-provided benefits received.

CPID Model Demonstration Project Objectives:

CPID projects had a shared process for clients entering the program. While each program had targeted different career pathways, and each client followed a unique pathway based on their “point of entry” the end goal was the same: Move along a career pathway toward competitive integrated employment in positions with competitive wages and employer-provided benefits.

Program Elements and Outcomes

• Enter career pathways in targeted industry or sector
• Engage in training and education
• Obtain industry-recognized postsecondary credential(s)
• Achieve competitive integrated employment
• Increase earnings and employer-provided benefits

The Nebraska and Virginia Career Pathway projects were intended to pilot strategies that would help states align with WIOA measure focused on wages, credential attainment and measurable skill gains. The following chart compares the WIOA performance indicators with the CPID performance indicators.

• WIOA places greater focus on quality of outcomes as opposed to quantity of outcomes
• Approach should allow for more accurate comparison of performance among core partners

 

WIOA Indicators of Performance	CPID Indicators of Performance
Employment Rate – 2nd Quarter After Exit	Number of Career Pathways Accessed/Created
Employment Rate – 4th Quarter After Exit	Employment Rate – 90 Days
Median Earnings – 2nd Quarter after Exit	Median Earnings – Per Quarter
Credential Attainment	Credential Attainment
Measurable Skill Gains	Measurable Skill Gains
Effectiveness in Serving Employers	Working with Employers to establish demand is integral to VA’s CPID Strategy.

CPID Model Demonstration Project Outcomes:

Data Collection

Process and outcomes analyses for the model demonstration evaluations included multiple data sources. These data sources included qualitative and quantitative data elements, which allowed for comprehensive documentation of the project implementation, client and business satisfaction with services, and program results.

VR agencies and their partnering evaluation units created and maintained data collection processes to support the project management and evaluation activities. The following table offers sample data sources and descriptions that could be adapted to meet your program’s goals.

Lessons Learned

Nebraska and Virginia shared some of the lessons learned in implementing the Career Pathways approaches in their VR agencies. Vital to the success of the program were buy-in from leadership and field staff. It was important to train staff early on and get them involved in the planning process (i.e. Virginia involved them in planning the academies). It was also important to keep the program on the counselors’ radar through presentations, informal check-ins, internal newsletters and alerts on the intranet.

Both found that it is imperative to establish tracking mechanisms straightaway.

Nebraska’s CPAP program was set-up as an external program. An advantage to this approach is the CPAP recruiters had more autonomy and were able to make decisions when they were working with businesses and other partners. They encountered a few challenges including the agency going into Order of Selection (OOS) midway through the program, which had implications on recruitment. They also encountered some initial glitches in getting the VR case management system and the CPAP system to interface and collect data.

Virginia found that lean manufacturing has decreased the accommodations necessary for individuals with disabilities. Technological advancements have made this the case in many industries. They learned a few lessons specific to their work with partners and sub-contractors. It is important to use of a unified message with all partners. They also learned that they needed to conduct bi-annual reviews of contracts with deliverables as part of the continuous quality improvement process. Another unexpected finding was that paid internships were a harder sell to counselors.

Both agencies shared insights regarding recruitment and engagement strategies. The younger clients respond better to social media and different technologies. They also found that the jobseekers required much more hand holding upfront during the career exploration and in making the decision whether this was the right time to pursue training. Nebraska and Virginia also found that business engagement strategies were the crux of the Career Pathways program being effective.

Sustainability Planning

When implementing a new program or initiative, sustainability planning should start early on during the planning phase and revisited as implementation changes are made. Issues to be addressed in the sustainability plan include how it will be integrated into the VR system and existing services, ongoing resources needed for continuation of services, and system alignment.

Integrated into the VR Services

Career Pathways adds a focus on career advancement as a VR service to jobseekers and incumbent workers. State VR agencies looking to implement Career Pathways will need to determine how it aligns with the agency priorities and their capacity and available resources to support ongoing efforts. How will they prioritize clients seeking career advancement services and supports over other employment needs such as job development? Can “Career Advancement Services” fall under a separate title similar to the current authority for Supported Employment Services? Adding Career Pathways as a service also has implications on outreach and marketing strategies. VR will need to change individuals with disabilities’ view of VR as a one-time service to a career-long engagement that builds on past successes. Marketing strategies will also need to highlight the collaborative aspects of this model so that jobseekers are the recipient of a seamless service delivery system. Career Pathways models use a multi-system approach to enable system alignment across partners. This collaborative approach insures equal access to high quality training leading to industry recognized certifications for individuals with disabilities.

In addition to expanding the services being offered to individuals with disabilities, as a dual-customer strategy Career Pathways also broadens the scope of services being offered to businesses to include advancement of incumbent workers with disabilities (generate new referrals to VR from businesses) and a pipeline of qualified jobseekers to backfill recently vacated positions when an individual advances in their career (see Nebraska CPAP project).

System Alignment

As noted earlier, Career Pathways aligns with the Workforce Innovation and Opportunity Act (WIOA). It builds on existing and new partnerships with Workforce Development entities. This means that there is an opportunity to create an effective services delivery system across programs. This can help eliminate duplicate services and pooling of resources. This makes ongoing communication and a continuous improvement process and integral part of sustainability planning.

In addition to maximizing existing resources, State VR agencies and partners should identify additional sources of funding and opportunities to expand their network. There are long-term economic benefits of including community organizations, foundation, and other entities in these partnerships.