Kosovo - Reliable Energy Landscape Project Evaluation

- v01: Edited, anonymous dataset for public distribution.

The objective of RELP is to reduce the gap between energy demand and supply, through three main activities: subsidies for energy efficiency in Kosovo (SEEK), district heating and metering (DHM), and independent power producers (IPPs). On the demand side, RELP will address two barriers to investments in energy efficiency: a general lack of consumer awareness about energy saving measures and their benefits, including the benefits of metering and other energy saving devices, and the inability of poor households to pay for energy efficiency measures. These objectives will be achieved by lowering energy use through piloting household investments in energy efficiency, metering existing district heating consumers, and switching new consumers to cost-effective district heating. On the supply side, RELP will aim to reduce barriers to renewable energy entrants to the market by stimulating the business environment. In the original design of RELP, the women in energy efficiency (WEE) activity was a sub-activity of SEEK. As of July 2020, it was decided to treat WEE as a stand-alone activity given its objective of promoting female employment in the energy sector.

The evaluation of RELP is organized around the MCC RELP theory of change whose overarching objective is to reduce the gap between energy demand and supply. The evaluation will test the causal links in the theory of change using qualitative and quantitative data to determine whether the activities and inputs outlined in the RELP theory of change affect short- and medium-term beneficiary-level outcomes. In order to understand if the main objective-level outcome was achieved, the RELP evaluation will include descriptive analysis of broad trends and demographics of project beneficiaries and a combination of performance and impact evaluations. The proposed evaluation design incorporates findings from evaluator's (IMPAQ) scoping mission and subsequent conversations with all relevant stakeholders. Quantitative components of both performance and impact evaluations will use primary data, administrative data, and monitoring and evaluation data. Qualitative components will incorporate information from document reviews, key informant interviews (KIIs), and focus group discussions (FGDs) and will provide additional context to understand the quantitative impacts. Furthermore, as part of qualitative data collection, IMPAQ will conduct women-only focus groups and ensure that other key marginalized groups are well represented to understand their experiences. Finally, the evaluation will also include the evaluation of the WEE activity, which is not linked to the main objective-level outcome of reducing electricity demand and supply, but instead focuses on increasing female employment in the energy sector. The research questions that will guide the evaluation of RELP activities are briefly outlined below:

1. Were the activities implemented as designed and were the outputs outlined in the project logic achieved?

2. Did the SEEK activity result in reduced electricity consumption in the intervention areas? What was the contribution of various components of the program toward any changes in energy consumption (technical assistance to key institutions, BC&O activities, installation of energy efficient equipment)? Were energy savings previously envisioned during preliminary baseline studies and energy audits achieved, and why?

3. Did the SEEK activity identify a cost-effective model for reducing household energy use that could be scaled up nationwide and what is the overall expected energy reduction?

4. Did the DHM activity result in a change in energy consumption in the intervention areas? What was the contribution of various components of the program toward any changes in energy consumption (technical assistance to key institutions, BC&O activities, and implementation of consumption-based billing)? How many new households shifted to district heating as a result of the DHM activity?

5. Was the IPP activity successful in reducing the perception of financial risk for investors and commercial lenders in the energy sector, and why? Did the IPP activity contribute to increasing the electricity supply in Kosovo? If so, by how much and why?

6. How effective were program activities at increasing women's employment in the energy sector and at increasing investments in energy efficiency among women entrepreneurs?

7. Did the program meet the stated objective of reducing the gap between supply and demand of electricity? How sustainable are critical outcomes of the program, and why?

We briefly detail below the evaluation approach for each activity under RELP:

1. SEEK (Subsidies for Energy Efficiency in Kosovo) Activity

HER (Household Energy Retrofits) sub-activity

The primary quantitative approach to estimate the causal impact of the HER sub-activity on electricity consumption using the quasi-experimental ITS method. This method will involve analysis of high-frequency (monthly) household-level administrative data on electricity consumption from the Kosovo Energy Distribution Services (KEDS). We will complement the ITS method with a pre-post approach using baseline and endline primary quantitative survey data to provide additional analyses on consumption from other energy sources. The evaluation will also use qualitative data through FGDs and KIIs to provide an in-depth understanding of behavior change. In addition to baseline and endline, we will collect qualitative data in the interim for continuous monitoring and to develop a deeper understanding of effects.

AER (Apartment Energy Retrofits) sub-activity

Similar to the HER sub-activity, we will use a combination of and ITS approach and a pre-post approach to understand the effect of this sub-activity on energy consumption in individual apartment units and the common areas of MABs. We will use an ITS approach to understand the impact on the main variable of interest: electricity consumption. The pre-post approach will complement the ITS analysis and provide evidence on the mechanisms of change. All quantitative findings in the AER sub-activity will be complemented with a qualitative case study approach, to provide more nuanced understanding of changes in energy consumption at the MAB level. Qualitative data will be collected at baseline and endline, in addition to three rounds of interim data for continuous monitoring. Finally, we will also conduct a CEA for AER, measuring cost-efficiency at the MAB/apartment level.

2. DHM (District Heating Metering) Activity

We propose to estimate the causal impact of the DHM activity on the key outcome of interest-district heating consumption-through the ITS method, using high-frequency (monthly) administrative data for each beneficiary household from Termokos. Additionally, we will use a pre-post approach that will utilize primary survey data collected at baseline and endline to understand the impact of DHM on consumption of energy from other sources, including electricity. Quantitative data collection will be accompanied with qualitative data collection at baseline, endline, and interim periods to understand how different program components affect district heating consumption after the installation of meters and thermal valves. Qualitative data will also help contextualize the quantitative findings.

3. IPP (Independent Power Producer) Activity

We will implement a process-tracing approach to assess the impact of the IPP activity. Our approach will be primarily qualitative, analyzing data collected through (a) document review and monitoring and evaluation data; and (b) semi-structured KIIs and small group interviews. We will track the progress, successes, and challenges of the major implementation steps. The timeframe of our evaluation is too short to examine whether the IPP activity ultimately contributes to increasing the electricity supply in Kosovo. However, we will assess whether the IPP activity was successful in reducing the perception of financial risk for investors and commercial lenders in the energy sector. We will also qualitatively assess whether investors and lenders are more willing and able to support IPP project development.

4. WEE (Women in Energy Efficiency) Activity

The implementation and effectiveness of this sub-activity will be analyzed primarily through qualitative data from semi-structured KIIs and FGDs. Semi-structured KIIs will include a short quantitative component as well. We will also use document review and monitoring and evaluation data. Implementation will be assessed by analyzing whether program targets and outputs have been achieved as per the initial plan, and if any identified challenges were overcome. Through semi-structured KIIs and FGDs with beneficiaries, the effectiveness of program activities will be assessed. Additionally, we will use secondary data to assess the program's influence on women's academic and employment goals and access to job opportunities in the energy sector. In the case of entrepreneur grants, we will determine whether grants led to more efficient use of energy, lowered energy costs, and improved profit margins.

Interrupted Time Series Study populations are:

1. Quantitative Data Collection

· Impact Evaluation (ITS approach):

o Households under HER sub-activity

o Individual apartment units under AER sub-activity

o Households under DHM activity

We propose to rigorously estimate the impact of the HER, AER, and DHM activities on the key outcomes of interest for RELP, i.e. electricity consumption and expenditure, through an ITS approach. The ITS approach estimates the causal impact of a specific program activity by analyzing administrative time-series data from the electricity utility before and after the program activity is completed, and by assessing to what extent household-level outcomes change immediately after the completion of the installation of energy efficiency retrofits/electricity meters, relative to a possible preexisting trend. The ITS approach will estimate impacts, by creating a longitudinal panel of households, and will aim to use monthly administrative data from January 2018 to May 2024.

· Performance Evaluation (pre-post approach):

o Households under HER sub-activity

o Individual apartment units under AER sub-activity

o Homeowner association (HOA) representatives under AER sub-activity

o Households under DHM activity

The ITS approach proposed above will only allow us to study the impact of the retrofits on one variable of interest i.e. electricity consumption and expenditure. In order to understand why electricity consumption would be impacted we would need data on additional outcomes. In this context, we will complement the ITS analysis with a pre-post longitudinal analysis of the same household and explore additional outcomes, such as substitution between energy sources, electricity reliability, and willingness to pay for energy efficiency retrofits, among other outcomes, which will not be possible to examine through administrative data. We will compare outcomes during endline primary data collection in September-October 2023, with those measured during baseline primary data collection, to provide information about the evolution of household-level outcomes.

2. Quantitative Data Collection

o SEEK activity: HER and AER beneficiaries, Millennium Foundation of Kosovo (MFK), Implementing contractor (IC), retrofit implementers, ancillary stakeholders including municipality leadership.

o DHM activity: District heating households and businesses, MFK, IC, Termokos, Energy Regulatory Office (ERO), and ancillary stakeholders.

o IPP activity: IPPs, MFK, ICs, Kosovo Credit Guarantee Fund, ERO, and banks.

o WEE activity: Women entrepreneurs, women interns and their supervisors, summer camp attendees and their mentors, MFK staff, and ICs.

Questionnaires for will be developed for both quantitative and qualitative data collection in English, and will be translated into Albanian and Serbian. After an initial review the questionnaires will be translated back into English by an independent translator with no prior knowledge of the survey. The back translation from the Albanian and Serbian versions will be independently reviewed and compared to the English original, and differences in translation will be reviewed and resolved in collaboration with the original translators.

1. Quantitative Data Collection

HER survey.

Data will be collected using a household survey for households under the HER sub-activity. For the first iteration of the HER sub-activity, (i.e. 500 households), the primary quantitative data collected will have two main goals: (1) to develop an understanding of the households participating in the program thereby enabling the IC to make adjustments to programming before the second iteration; and (2) to conduct the cost-effectiveness analysis, for iteration 1 to create a measure for the cost-effectiveness of the payment plan chosen for the first iteration. As per IMPAQ's understanding, there is no endline survey envisioned by the IC for the first iteration beyond the household satisfaction survey. For our main sample for the HER (i.e. 2,100 households in the second iteration for HER), we will use primary data collected by in-person household surveys at two points in time: baseline and endline to estimate mechanisms of change for our main outcome variable (i.e. electricity consumption). Specifically, primary data collected through the baseline and endline survey will be important to understand the substitution between different energy sources, the mechanisms of change, and the impact of the BC&O campaign. The main respondent of this survey will be the household head. The goal of the baseline data collection will be to obtain pre-intervention values of our main outcome variables. The goal of the endline data collection will be to quantify the change in energy usage behavior for each household. The survey will include detailed questions about the demographics, physical characteristics of the house, and different sources of energy and cost of each source, among other outcomes. The IC will collect the baseline data, with IMPAQ's support. At baseline, the IC will procure the data collection firm and IMPAQ will lead the enumerator training, survey instrument development, pilot testing, and quality checks. At endline, IMPAQ will be responsible for procuring the data collection firm and leading the data collection efforts.

AER survey.

Data will be collected using a similar household survey for individual apartment units under the AER sub-activity. In-person household surveys at two points in time: baseline and endline to estimate mechanisms of change for our main outcome variable (i.e. electricity consumption). Specifically, primary data collected through the baseline and endline survey will be important to understand the substitution between different energy sources, the mechanisms of change, and the impact of the BC&O campaign. The main respondent of this survey will be the household head. The goal of the baseline data collection will be to obtain pre-intervention values of our main outcome variables. The goal of the endline data collection will be to quantify the change in energy usage behavior for each household. The survey will include detailed questions about the demographics, physical characteristics of the house, and different sources of energy and cost of each source, among other outcomes. The IC will collect the baseline data, with IMPAQ's support. At baseline, the IC will procure the data collection firm and IMPAQ will lead the enumerator training, survey instrument development, pilot testing, and quality checks. At endline, IMPAQ will be responsible for procuring the data collection firm and leading the data collection efforts. The AER survey will have two modules: (1) the first module will be similar to the HER survey and will include questions to understand household level decision-making; and (2) the second module will be limited to those apartment units whose household head is part of the HOA association of the MAB to understand behavior change and decision-making at the MAB level.

DHM household survey.

We will use data collected by in-person household surveys at two points in time: baseline and endline. The IC will be responsible for baseline data collection and IMPAQ will be responsible for endline data collection. The main respondent of this survey will be household heads. The goal of the baseline data collection will be to obtain pre-intervention values of our outcome variables including consumption of energy from other sources, demographic characteristics, and physical characteristics of the apartment. The goal of the endline data collection is to collect information similar to the baseline survey and also collect additional information about the experiences with CBB, new tariffs, and perceptions about its effectiveness. Similar to the HER and AER sub-activities, the baseline data will be collected by the IC, with IMPAQ's support. At baseline, IMPAQ will lead data collection efforts including enumerator training, survey instrument development, and pilot testing and the endline data will be collected by IMPAQ.

Administrative data.

HER and AER sub-activities. This data will be collected for beneficiary households in the HER and AER sub-activities. The administrative data will also allow us to measure outcomes on a continuous basis. KEDS data will also provide us reliable information about our main outcome variables: monthly electricity consumption and expenditures. Additionally, it will provide historical electricity consumption information for each household. IMPAQ will be responsible for coordinating with KEDS to obtain necessary administrative data. During the baseline in-person surveys, we will ask households for their KEDS invoice number, so that we can track their electricity usage from the KEDS database. We will collect historical data for each household from January 2018 onward and until the end of the evaluation period in May 2024. We will execute a data-sharing agreement with KEDS, under which they will transmit data to IMPAQ periodically.

DHM activity. Administrative data will be collected for beneficiary households sampled under the IC's baseline survey. The goal of collecting administrative data will be to measure our main outcome variables (i.e. district heating consumption and expenditure) on a continuous basis. We will obtain administrative data from both Termokos and KEDS, since we expect that the households on the district heating network will also have a KEDS connection. The administrative data from Termokos will provide information about the monthly district heating consumption, and administrative data from KEDS will provide monthly data about electricity consumption, if any. IMPAQ will be responsible for coordinating with Termokos and KEDS to obtain necessary administrative data. During the baseline in-person surveys, we will ask households to provide their Termokos invoice number and their KEDS invoice number and will also obtain consent to access their data. We will initiate a data-sharing agreement with Termokos and KEDS to transmit data to IMPAQ on a monthly basis in addition to historical data as requested by IMPAQ.

2. Qualitative Data Collection

HER sub-activity. Our data collection will include FGDs with both beneficiary and “non-beneficiary” households. We define non-beneficiary households as those households who reside within the same community and beneficiaries. From the non-beneficiary households, we can learn why households that were exposed to the same outreach campaigns decided not to participate, to determine whether any adjustments can be made to the HER sub-activity to encourage future enrollment. For example, it may emerge that lower-income households do not understand the benefits, including the ultimate cost savings, of participating in the program. Alternatively, we may find that these households are very aware of the benefits and would like to participate, but are unable to navigate the financing process and/or pay the necessary costs up front. Once implementation for the HER sub-activity is completed and we have preliminary findings, we will hold FGDs with a mix of beneficiary households with both high and low energy consumption, to examine underlying behavior and contributing factors. We will also hold KIIs with associated stakeholders, including qualified installers. Finally, we will examine the extent to which gender was incorporated by the IC at various implementation stages including selection of beneficiaries, design of grant levels, procurement of qualified installers, BC&O campaign, capacity building, and installation of retrofits.

AER sub-activity. For the AER assessment, we propose a process-tracing method with seven single case studies. The unit for the case study is the MAB, and we will select one MAB per municipality. We will ensure that our cases represent special beneficiary target groups, including poor/low income, disabled/seriously ill, female-headed (including single mothers and elderly women), and minorities. Within each case study, we will develop FGD or small group interview with apartment residents and HOA representatives (or alternative leadership). As part of the case studies, we will interview those who led the MAB retrofitting design and construction, which includes the corresponding designers, contractors, and supervisors. With guidance from MFK and MCC, we will interview any other ancillary stakeholders as appropriate and feasible, such as municipality leaders, if the municipalities invested in the MABs as planned. We will incorporate SEEK monitoring, evaluation, and learning data and our own quantitative data in the individual case studies. Finally, we will examine the extent to which gender was incorporated by the IC at various implementation stages of AER including selection of MABs, design of grant levels, establishment of HOAs, building audit and program design, procurement and contracting, BC&O campaign, capacity building, and installation of retrofits at the MAB level.

DHM activity. We will use FGDs to get the perceptions and experiences of district heating customers about the transition toward a CBB system, as part of the DHM activity. We will include households with a mix of characteristics to develop insights into the contributing factors towards behavior change. To holistically assess the DHM activity, we will also hold KIIs and small group interviews with representatives from Termokos, the ICs, ERO, MFK, and any other associated stakeholders. The goal will be to assess and contextualize program activities, including those related to preparing institutional, organizational, and regulatory measures; setting end-user tariffs; and designing new billing and collection procedures.

IPP activity. We will conduct KIIs with a wide variety of stakeholders at five points during the evaluation. During baseline data collection (round 1), we will conduct site visits to interview IPPs that plan to apply for financing under this activity. We will also hold KIIs with businesses that plan to use the project to buy solar panels for private consumption. During our subsequent rounds of data collection, we will include participating IPPs and any IPPs who originally participated but have dropped out. Including nonparticipating IPPs will be useful to get the perspective of those who were unsuccessful in becoming bankable. This will allow us to determine how the renewable energy project in Kosovo is progressing after the completion of capacity-building activities. This may also provide enough time to examine early outcomes, such as the status of construction undertaken by some IPPs.

WEE activity. Data for the WEE activity will be collected using semi-structured KIIs and FGDs, as well as through review of existing records and documentation and post-activity participant surveys. The main questions for the qualitative data collected will cover perception change, project success, sustainability, experience with the project, and project outcomes at the sample unit level. To evaluate project effectiveness and sustainability, a number of records will be collected and reviewed at baseline and end line. For example, for the entrepreneurial grants activity, cash flow and profit and loss statements from entrepreneurial entities will be collected and analyzed for evidence on increase investments in energy efficiency measures or increased profits related to decreased energy expenses. For the internship activity, participating company human resource records will be analyzed at baseline and endline to know whether or not proportional hiring of women has increased, if women serving internships have been hired, or if guidelines for hiring have been adapted to facilitate increased hiring of women. Summer camp applications and attendances will also be analyzed to note an increase over time, which will suggest sustainability of this effort. Finally, post-activity surveys will be used to solicit information as to whether women using scholarships or participating in internships have gained employment in STEM sector fields within two years after their WEE experience. This evidence will provide insights as to the outcome of these interventions. Likewise, follow up with summer camp participants may shed light on the continued interest and pursuit of STEM-sector fields after summer camp activities.

HER sub-activity: National coverage

AER sub-activity: Coverage limited to seven municipalities including Pristina, Lipjan, Novobrdo, Gracanica, Gjakova, Mitrovica, and Viti

DHM activity: Only in capital city of Pristina

IPP activity and WEE activity: National coverage