El Salvador - TVET System Reform

I. Introduction

The Human Capital Project is one of three projects under the second compact, and it focuses on improving the quality of education and achieving a better match between the supply of skills and the demands of the labor market as they pertain to the international trade of goods and services in El Salvador. The Human Capital Project consists of two activities: (1) the Education Quality Activity, which is intended to improve the quality of the national education system; and (2) the Technical Vocational Education Training (TVET) System Reform Activity. The TVET System Reform Activity, with funding of nearly $15 million, is intended to enhance its capacity to harmonize the skills supplied through education and training providers with the skills demanded by the labor market.

MCC contracted with Mathematica to conduct an evaluation of the activities under the Human Capital Project, including the TVET System Reform Activity. Mathematica will conduct a performance evaluation of the TVET System Reform Activity to document its implementation and assess how successful the project was at achieving the intended outcomes. Lessons learned through this performance evaluation will be useful for stakeholders as they continue improving the TVET system, and to inform decisions about the potential expansion of the reform activities to other sectors beyond the sectors focalized by FOMILENIO II.

II. Overview of the Technical Education and Professional Training System (TEPT) Activity

The revised implementation of this Activity encompasses two main sub activities:

1. Integrated Technical Education and Professional Training Systems Sub-Activity: The revised implementation plan for this sub-activity includes two strategies:

a. Capacity building at the individual, organizational, and public policy level. The capacity building strategy has two goals: the first is to enable the private and public sector to identify and manage demands for technical education and professional training. The second is to enable public sector stakeholders involved in the design and implementation of education and training programs to respond quickly and efficiently to private sector labor market demands.

b. Strengthen the link between the supply and demand of technical education and professional training through the establishment of mechanisms that facilitate the exchange of information within and between productive sectors , and between private and public institutions in all productive sectors. FOMILENIO II will support the development of sectoral diagnostics to identify the needs of the productive sector, and improve the match between the industries and the training offerings in specified geographic areas. The following are the main tasks for implementing this strategy: 1) identify the prioritized productive sectors in each department and the availability of related technical and professional programs in those geographic areas; 2) design or update three optional modules of the BTVs with input from the Sectoral Committees; and 3) Establish agreements between companies from the productive sectors and MINED to offer internships for students.

2. Continuous Labor Demand Assessment Sub-Activity. Two strategies encompass the implementation of this sub-activity: 1) establish a monitoring and evaluation (M&E) framework; and 2) strengthen the labor market observatory, integrating existing efforts to track labor market conditions.

III. Logic Model

The TEPT Activity in El Salvador is designed to link activities in the systems of technical education and professional training to the needs of the labor market in order to close the gap between training and employment. To that end, TEPT activities are largely focused on facilitating the exchange of information between the supply side and the demand side, both within and between productive sectors. The theory of change behind the implementation of the TEPT Activity states that all of its outputs will increase companies' productivity, and thereby improve workers' wages as well as the employability and socioeconomic standing of students and workers who benefit from the TEPT Activity.

1. Activities

The TEPT Activity funded by FOMILENIO II aims to strengthen capacity building at the individual, organizational, and public policy level. To that end, this strategy includes (1) creating sectoral committees for technical and educational training, and creating a Sectoral Committee Board to consolidate the demand for professional development in the private sector; (2) offering professional training to workers in the different industrial sectors under the competency-based approach; (3) designing models for certifying workers by competencies and accrediting institutions; (4) creating a Coordination Council of Technical Education and Professional Training to facilitate coordination between public and private organizations involved in technical and professional training, and align the offerings to the skills demanded in the labor market; and (5) establishing procedures and regulations to strengthen the institutional capacity to meet the private sector's demand for technical education and professional training. Implementation of the TEPT Activity includes the design of a roadmap for creating a national qualifications framework, and the development of an information system for the Coordination Council and other bodies to use in the planning, monitoring, and evaluation of the TEPT Activity.

The TEPT activity also seeks to improve the link between labor demand in the productive sectors with the availability of technical education and vocational training. To achieve this goal, FOMILENIO II plans to: (1) identify the prioritized productive sectors in each department and the availability of related technical and professional programs in those geographic areas; (2) design at least three modules for the Vocational Technical Baccalaureate in coordination with the Sectoral Committees; and (3) establish agreements between companies from the sectors and MINEDUCYT to offer internships for students.

2. Short- and medium-term outcomes

Once the sectoral committees and the Board of Sectoral Committees are in existence, and a model for certification and accreditation has been implemented, the private sector will improve its capacity to identify and influence the professional training that is available to prospective trainees. Designing modules for the Professional Technical Baccalaureate, and creating the Coordination Council of TEPT will also improve the linkage between the demands of the productive sector, and the availability of technical education and vocational training.

3. Long-term outcomes

In the long term, the TEPT Activity is expected to improve the government's capacity to plan and evaluate the offer of technical education and professional training assisted by the MNC and the information system developed for subsequent planning, monitoring, and evaluation. Finally, the improved match between labor demand in the industrial/private sectors and the technical training available to the potential workforce will improve the productivity of companies, and therefore increase workers' wages. It is also expected to improve the employability of students and workers who benefit from the TEPT Activity.

IV. Evaluation design

The aims of the performance evaluation are to (1) assess progress toward the short- and medium-term outcomes that the program was designed to achieve and (2) understand and document the implementation of the Technical Education and Professional Training activities. To meet these two overarching aims, the performance evaluation will include a pre-post analysis of program outcomes and an implementation study.

A. Pre-post design.

The single group pre-post design consists of obtaining data for the outcomes of interest prior to (or at the beginning of) TEPT training (pre) and the same measure after participation in the TEPT courses (post). The pre-post study will focus on measuring the changes on labor market outcomes, and assessing to what extent work conditions, wages, and employment status improved for those who participated in the TEPT programs.

B. Implementation study

The implementation study will focus on describing all components of the TEPT program and their operation and identifying progress towards short- and medium-term outcomes of the program. We will document how program components were implemented, explore the main facilitators and barriers to the implementation of TEPT sub-activities, and describe the main program outputs of the TEPT System. We will also identify short- and long-term achievements and sustainability efforts related to each TEPT sub-activity.

V. Evaluation research questions

The pre-post study will focus on three descriptive research questions:

1. Did employee work conditions, benefits or non-wage compensation, change for trainees who participated in the TEPT courses approximately six to nine months after the courses ended?

2. Did wages change for trainees who participated TEPT courses (as measured about six to nine months after the courses ended)?

3. Did employment status change for trainees who participated TEPT courses (as measured about six to nine months after the courses ended)?

We will examine five main overarching research questions through the implementation study:

1. How were the TEPT sub-activities implemented, and what were the major facilitators and barriers to implementation?

2. Were some sectors more successful implementing the TEPT activities than others?

3. Which outputs were accomplished for each sub-activity, and to what extent did they meet the targets?

4. Which short- and long-term outcomes were accomplished for each sub-activity?

5. What are the plans for the institutional and financial sustainability of the services offered and the entities created as part of the TEPT activity?

VI. Data sources and analysis plan

A. Pre-post study:

Data sources.

The primary data source for the performance evaluation will be the trainee tracer survey. In collaboration with FOMILENIO II, we will field the pre-training tracer survey (baseline) with about 500 participants who are enrolled in TEPT courses, either prior to or within one month of their course initiation. TEPT courses will be rolled out starting the second quarter of 2019. Because the timing for the rollout will vary, baseline data collection will take place on a rolling basis and will be aligned with the course rollout plan for the prioritized sectors. We plan to conduct the baseline survey as a Computerized self-administered questionnaire (CSAQ) with groups of enrolled trainees at their training sites and the follow-up survey as an individual, face-to-face, computer-administered questionnaire at respondents' work sites or homes. We will collect the follow-up survey six to nine months after the course ends by following a staggered timeline similar to the baseline data collection. The follow-up survey is expected to start in mid-2020 and to extend beyond the compact period.

Analyses.

To assess changes in work conditions, wages, and employment status sometime after participants should have completed the TEPT training programs, we will conduct a descriptive analysis of the tracer survey data for trainees in FOMILENIO II-supported courses. We will compare key outcome indicators at baseline and follow-up for each trainee-level outcome and report averages (mean percent change) for the full sample of trainees and also by gender. We will also describe the background trainee characteristics, such as gender, age, and education level. The analysis will include a cross-sector comparison of participant characteristics and implementation milestones.

B. Implementation study

Data sources.

We will draw on three sources to obtain qualitative and quantitative information on outputs and achievements of the TEPT activity: (1) stakeholder interviews, (2) program monitoring data, and (3) a document review.

Analyses.

a. Analytic approach for quantitative program implementation data. We will conduct a descriptive statistical analysis on monitoring data to summarize the program outputs and understand short-term and long-term program achievements.

b. Analytic approach for qualitative program implementation data. We will prepare interview and focus group transcripts and conduct thematic analysis in two sequential steps. The first step involves conducting data reduction and mapping transcript segments to the core performance research questions. The second step involves thematic framing-the process of distilling themes and identifying patterns in qualitative data. We will develop a coding framework with a hierarchy of conceptual categories that are linked to the research questions and the program logic model. We will also create thematic memos for data synthesis and interpretation of key findings.

To take advantage of the complementarity that a mixed-methods approach affords, we will integrate findings from descriptive quantitative analysis of monitoring and administrative data with findings from qualitative data sources. We will summarize program outputs and outcomes from FOMILENIO II's administrative records and implementation reports by using data abstraction templates that we will design for this study. We will also create summary tables of qualitative findings from key stakeholder interviews. Then, we will integrate the quantitative and qualitative findings by using triangulation techniques to (1) assess the consistency of the findings across methods and data sources, (2) confirm patterns of findings, and (3) identify potential discrepancies across data sources. Furthermore, information from the qualitative data collection related to barriers and facilitators to program implementation will help us interpret the findings from the quantitative analysis.

Other (Performance Evaluation) Individuals, institutions/organizations, groups.

TEPT activities are being implemented nationally in El Salvador.