Jordan - Water Infrastructure

- v02: Edited, anonymous dataset for public distribution, expanded to include water meter testing, non-revenue water testing, and billing and consumption datasets provided by utility

Evaluation Design

MCC contracted Social Impact (SI) to design and implement an independent, rigorous evaluation of Compact programming in September 2012. The evaluation uses a mixed design, blending impact evaluation, performance evaluation, cross-sectional survey research, predictive modeling, and tracking of performance measures to answer evaluation questions. The evaluation used six primary methodological approaches:

M1: Quasi-experimental matching + difference-in-difference analysis of panel data (IE): To measure JC impacts on beneficiaries in Zarqa, the IE drew on data from a representative panel survey households and enterprises who were differentially exposed to the infrastructure improvements. Use of controls from Zarqa and neighboring East Amman estimate utility-wide spillovers from network rebalancing.

M2. Natural experiment + difference-in-difference analysis of panel data (IE): To measure outcomes among farmers whose supply and source of water was affected by the JC, the evaluation used a representative panel survey combined with a natural experiment of different agricultural zones that experienced a varying shift towards blended Zarqa River water during the course of the study.

M3. Water balance modeling projection of impacts of JC investments through 2050 (model-based IE): The evaluation used a detailed “with” and “without” Compact water balance analysis to predict changes in water flows as a result of the JC, projecting from 2012 to 2050. This analysis uses the most recent version of MWI's Water Evaluation and Planning Tool (WEAP), as well as secondary data collected with the help of MWI and MOPIC. JC contributions are included through reductions in the physical water losses and increased wastewater flows, with the “without” counterfactual assuming no changes.

M4. Utility performance tracking (PE & IE): To assess benefits to the Zarqa water utility, the evaluation compared utility performance indicators over time in Zarqa against those in Amman and Aqaba. Additionally, the evaluation conducted meter accuracy tests before and after the JC, and drew on the IE framework to conduct zonal analysis of billing and maintenance events in the WNP and matched comparison areas.

M5. Complementary cross-sectional assessments that provide information on contextual and confounding factors: To control for confounding factors that emerged over the 7 years of the evaluation, SI used cross-sectional assessments of 1) refugee settlement and water consumption; 2) water vendors in Zarqa and Amman; 3) water meter error; 4) non-revenue water (NRW) in JC DMA's; and 5) Qualitative Key Informant Interviews (KIIs) with important stakeholders involved in Compact design and implementation.

M6. Focused econometric analysis: During baseline surveys, the contingent valuation method was used to allow for valuation of potential reliability increases among households and farmers. SI uses these data to answer questions about the relative value of changes in water supply as perceived by customers.

Evaluation Question and Exposure Period

Impacts on compliance: Does the AEP result in increases in the quantity of wastewater that meets effluent standards prior to discharge into the environment? Total exposure: 40 months

Does the combined WNP/WWNP/AEP result in increased irrigation with blended KTR water? Does the volume of irrigation using KAC freshwater correspondingly decrease? Total exposure: 27-50 months

Does the combined WNP / WWNP / AEP lead to changes in farm input costs? Total exposure: 27-47 months

Does the combined WNP / WWNP / AEP lead to changes in the value of farm output in affected areas? Total exposure: 27-47 months

Are farm values affected by the WNP / WWNP / AEP investments? Total exposure: 27-47 months

What is the net economic value of changes in irrigation? Total exposure: 27-47 months

What reallocation of water is made possible by the Compact investments? ('primary' substitution effect)? Total exposure: 31-50 months

Impacts on environmental quality: Does the WNP alter the quality of water consumed at the HH / E levels? Does the WWNP reduce the risk of disease from exposure to untreated wastewater? Total exposure: 27-47 months

Does the WNP change the quantity of water consumed at the household (HH) and enterprise (E) levels (reduced leaks, increased reliability)? Total exposure: 27-38 months

Does the WNP affect time and money expenditure on water ('secondary' substitution effect)? Does the WWNP change consumer expenditure on wastewater management and disease prevention and treatment? Total exposure: 27-47 months

Does the net cost recovery of the utility improve due to the Compact, and is this related to service improvements? Total exposure: 31-50 months

What is the impact of the Compact on the budget and execution of O&M? Total exposure: 31-50 months

At the utility level, are there measurable changes in service delivery quality trends in Zarqa relative to those of other municipal utilities in Jordan? Total exposure: 31-50 months

Does the WWNP change consumer expenditure on wastewater management and disease prevention and treatment? Total exposure: 27-47 months

Does the WNP change HH / E income? Total exposure: 27-38 months

Does the WNP / WWNP affect property/asset values? Total exposure: 27-47 months

What is the net economic value of changes in quantity and quality of water consumed? Total exposure: 31-50 months

What is the comparative economic value of water consumption for different uses (domestic, commercial/ industrial, irrigation)? Total exposure: 31-50 months

What are the overall net benefits from the Compact, and what are its distributional consequences? Total exposure: 27-53 months

Summary of Results

Impacts on compliance: Does the AEP result in increases in the quantity of wastewater that meets effluent standards prior to discharge into the environment? Yes; flows to the AS WWTP increased by about 18 million m3 per year from 2014-2018 due to general population growth and increased wastewater connections in Zarqa; effluent release has been fully compliant with Jordanian standards.

Does the combined WNP/WWNP/AEP result in increased irrigation with blended KTR water? Does the volume of irrigation using KAC freshwater correspondingly decrease? Yes; the WNP and WWNP investments contributed 4.4-5.5 million m3 per year to the increased inflow to the AS WWTP, an amount that may increase to about 7.5 million m3 by 2050. The treated effluent is blended with Zarqa River water, and much of it (minus some withdrawal in the highlands along the Zarqa River) then reaches the King Talal Reservoir and then farmers downstream in the Jordan Valley. This additional flow is increasingly displacing KAC freshwater in the North JV, where farmers report a 10-15 percentage point increase in the relative sourcing of blended to freshwater sources.

Does the combined WNP / WWNP / AEP lead to changes in farm input costs? Not overall, but the composition of input costs changed. There are no consistent changes in total input costs in areas using increasing blended water as a result of the Compact, but vegetable crop inputs are increasing while tree crop inputs are decreasing, relative to locations that consistently used only blended water since before the Compact. Overall input costs have increased in the highlands downstream of the AS WWTP relative to other areas, likely because increased water availability from the Compact has led to increased production along the Zarqa River.

Does the combined WNP / WWNP / AEP lead to changes in the value of farm output in affected areas? Not overall, but composition of output changed. There are no consistent changes in total revenues among farms which received more blended water as a result of the Compact. However, there was variation between specific crops: farms which received more blended water saw increases in the value of their vegetable output, alongside decreases in the value of tree crop output, relative to farms which had consistently received only blended water from before the Compact and experienced no change in water supply. Among farms in the highlands downstream of the AS WWTP, overall output and profits increased - consistent with their increased production.

Are farm values affected by the WNP / WWNP / AEP investments? Yes; there is some evidence of relative improvements in land values in areas receiving increased blended water, relative to locations that had consistently received only blended water prior to the Compact. This may be due to a perceived improvement in the long-term sustainability of agriculture there as flows of blended water are anticipated to increase.

What is the net economic value of changes in irrigation? Farms in the JV that experienced an increased supply of blended water have sustained their economic value, with no net profitability change relative to comparison areas where the supply stayed constant over the evaluation period, while farms in the Highland saw profits increase. Given the fact that flows of blended water increased in the North JV, the lack of changes in relative profitability there suggest that the value of farming increased relative to a hypothetical no-Compact counterfactual with less water, and relative increases in land value in the northern JV suggests that perceptions of the sustainability of farming in areas supplied by freshwater prior to the Compact have improved.

What reallocation of water is made possible by the Compact investments? ('primary' substitution effect)? Our analysis suggests that 4-6 million m3 per year of additional water supply (net of water losses) reached urban consumers (mostly in Zarqa) between 2014 and 2018, an amount that may increase to 11 million m3 per year in the period up to 2050.

Impacts on environmental quality: Does the WNP alter the quality of water consumed at the HH / E levels? Does the WWNP reduce the risk of disease from exposure to untreated wastewater? No; there is no consistent evidence of changes in objective or subjective measures of household-level water quality in WNP areas; relative to stored water, including from shops, tap water quality remains good and unchanged from baseline, even as perceptions remain pessimistic about the relative quality of network versus vendor (shop or tanker) water. We see inconsistent results for disease (mixed evidence of increases in WWNP areas), but our samples are not powered to detect such changes, which suggests they are likely spurious.

Does the WNP change the quantity of water consumed at the household (HH) and enterprise (E) levels (reduced leaks, increased reliability)? No; there is only weak evidence that expenditures on vendor and network water may have declined, especially relative to controls in Amman. The latter changes cannot be clearly attributed to the Compact, since quantities consumed of shop and tanker water did not decline. There is no evidence of time savings from improved water supply.

Does the WNP affect time and money expenditure on water ('secondary' substitution effect)? Does the WWNP change consumer expenditure on wastewater management and disease prevention and treatment? No; there is only weak evidence that expenditures on vendor and network water may have declined, especially relative to controls in Amman. The latter changes cannot be clearly attributed to the Compact, since quantities consumed of shop and tanker water did not decline. There is no evidence of time savings from improved water supply.

Does the net cost recovery of the utility improve due to the Compact, and is this related to service improvements? Yes, but only in the short term; operating cost recovery initially improved sharply in 2014-2016 (from 60 to close to 90 percent) as revenues increased with sewer subscriptions and meter replacement. Despite decreasing NRW, cost recovery has since declined (back to about 70 percent), likely due to increased energy cost, higher maintenance cost, and the cost of additional wastewater management. The only clear link between cost recovery and service quality is in the revenue increase from higher wastewater connection rates.

What is the impact of the Compact on the budget and execution of O&M? Operating costs per unit of water supplied have nearly doubled over the period 2014-2018, though these changes are partly due to non-Compact drivers. Planned energy savings were not realized, likely due to rising energy costs, continued pumping of water to many WNP DMAs, and an increase in pumping of sewage. Pipe bursts overall have decreased since 2016, after an uptick in late 2015 and early 2016. In addition, anecdotal evidence indicates an improvement in response times to customer complaints and leaks. Network maintenance actions, meanwhile, have been stable overall due to reallocation of maintenance resources; these sharply decreased in WNP areas in 2017 and 2018 but were matched by a correspondingly sharp increase in non-Compact areas, likely reducing overall service interruptions and improving service to customers.

At the utility level, are there measurable changes in service delivery quality trends in Zarqa relative to those of other municipal utilities in Jordan? Yes; there is some evidence of service quality improvements relative to other utilities. The volume of water delivered per customer increased by 5 percent in 2015-2018 in Zarqa, but was flat or declined in other utilities. “No-Water” complaints have declined by 7 percent even as other utilities have experienced increases. From the household surveys, there is strong evidence of service quality improvements in WNP areas (continuity of service and pressure at the household level, and declining customer complaints).

Does the WWNP change consumer expenditure on wastewater management and disease prevention and treatment? No; contrary to expectations in the pre- and post-Compact ERR analysis, there is no evidence of savings on wastewater management thus far, though this impact may take longer to materialize given the infrequency of cesspit maintenance. Because more people are connected to sewers in WWNP areas, the average expenditure of sewer connections across all households (including those who were not newly connected) increased. However, the cost paid by each household that connected was lower than the cost for households in non-WWNP areas. We see inconsistent results for disease (mixed evidence of increases in WWNP areas), but our samples are not powered to detect such changes, which suggests they are likely spurious.

Does the WNP change HH / E income? No; there is weak evidence that the WNP may have decreased household or enterprise income or revenues, though it is difficult to explain (based on results leading up to this change) how this would have occurred, which suggests that it may represent a secular increase in incomes in Amman relative to Zarqa.

Does the WNP / WWNP affect property/asset values? No; there is no evidence that the investments affected household or enterprise land or asset values.

What is the net economic value of changes in quantity and quality of water consumed? Unclear. While metered consumption clearly increased in Zarqa, a substantial portion of this change is due to reduced meter error. Water-related expenditures by households and enterprises did not change due to the JC. However, additional water consumption for households would deliver modest quality of life benefits, as measured in stated “willingness to pay” questions. Such benefits arise from enhanced reliability and water consumption, but are mitigated by the network intermittency. For the utility, the additional supply does displace more expensive supply alternatives (e.g., from Disi or other supply augmentation investments), leading to net substantial cost savings at the system level. This cost is estimated at 0.7-1 JD per m3 (alternative cost for Disi supply). Assuming that 1 JD per m3 is appropriate considering the cost of future new projects, the implied savings will range from about 5-8 million JD per year through 2050. There is no economic value from changes in water quality, which was unchanged by the Compact.

What is the comparative economic value of water consumption for different uses (domestic, commercial/ industrial, irrigation)? Willingness to pay for additional water, the measure of its consumption value by various consumers (domestic - households, irrigation - farmers, commercial/industrial - enterprises), was found to be generally low in surveys. An alternative definition of economic value, based on alternative costs avoided, suggests that the value of water to domestic and commercial or industrial consumers is highest, since the next-best alternative for such urban consumers is expensive Disi water.

What are the overall net benefits from the Compact, and what are its distributional consequences? The evaluation results largely support the revised analysis in the closeout ERR, suggesting that overall net benefits are positive and higher than expected at signing for the WWNP/AEP (due to an increased value for primary substitution), but lower for the WNP (due to weak evidence for secondary substitution). These assessments are based on differences in the timing and scale between the results of the evaluation and what was expected for the various benefit streams in that cost-benefit analysis (CBA) work, though we highlight that the additional benefit streams may perhaps materialize in the future.

Regarding distributional consequences, urban consumers and the utility appear to have benefited the most. For urban consumers, the primary benefit has been from improved water supply service. The utility has primarily benefited from decreased NRW, offsetting increases in operating costs. Farmers have also benefited from sustained water supply, with no net adverse effects on profits from increased use of treated wastewater.

Major variables and data sources:

1) Water consumption and management habits (Households, farmers, enterprises, Miyahuna billing database)

2) Expenditure, asset ownership, income, profit (Households, farmers, enterprises)

3) Health outcomes (Households)

4) Water throughput (Water meter testing, non-revenue water tests)

Propensity Score Matching; Difference-in-Difference; Pre-Post with Comparison Populations; Other Households, enterprises, farms, other (meters/DMA)

Sample survey data [ssd]; Administrative records data [adm]; Observation data/ratings [obs]

Household Survey - panel survey conducted in Winter-Spring 2014, Winter 2015, Summer 2015, Winter 2016, and Summer 2019. Sample of 3440 households was drawn at the census block (“geocode”) level, with matched treatment and control blocks according to baseline census characteristics. Survey contents included household socio-economic status, water consumption, sourcing, storage, and treatment, expanded in the endline to include knowledge of illegal usage of water in the area, and randomized response questions around personal illegal usage or connections to water network. Questionnaires were prepared in English by SI, translated into Arabic, and tested prior to each round of the survey for comprehension. The results were back-translated into English. Conducted in Arabic using Android Tablet. All questionnaires are provided as external resources.

Agriculture Survey - panel survey conducted in Spring/Summer 2015, 2016, and 2018 with 550 farms in the Jordan Valley and Highlands. Contents included cropping patterns, expenditures and incomes, assets, and water usage - sourcing, expenditure, quality perceptions. Expanded in the endline to include knowledge of illegal usage of water in the area, and randomized response questions around personal illegal usage or connections. Questionnaires were prepared in English by SI, translated into Arabic, and tested prior to each round of the survey for comprehension. The results were back-translated into English. Conducted in Arabic using Android Tablet. All questionnaires are provided as external resources.

Enterprise Survey - panel survey conducted in Spring 2014 and Spring-Summer 2018, using a Department of Statistics list of small- and medium-enterprises for initial selection of respondents, within the same geocodes used for household sampling (Zarqa & Amman). This list did not give the targeted sample of 425, and was later supplemented with references from households and direct recruitment by enumerators on the ground. Contents included enterprise's economic status and activity, expenditures and income, and water consumption, sourcing, and treatment costs. Expanded in the endline to include knowledge of illegal usage of water in the area, and randomized response questions around personal illegal usage or connections. Questionnaires were prepared in English by SI, translated into Arabic, and tested prior to each round of the survey for comprehension. The results were back-translated into English. Conducted in Arabic using Android Tablet. All questionnaires are provided as external resources.

Vendor Survey - Tankers - cross-sectional survey conducted in Fall 2018, using ad-hoc sampling that combined a list of registered tankers from department of Motor Vehicles with exhaustive recruitment at filling areas and tanker parking areas (traffic circles where customers place orders). Coverage is believed to include all tankers active in Zarqa and East Amman. Content includes water sourcing, treatment, business characteristics (number of employees and tankers, income, expenditure), sales and shifts in customer habits or number, and external factors influencing their business including regional shifts in water supply. Questionnaires were prepared in English by SI, translated into Arabic, and tested prior to the survey for comprehension. The results were back-translated into English. Conducted in Arabic using Android Tablet. All questionnaires are provided as external resources.

Vendor Survey - Water Shops - Cross-sectional survey conducted in Fall 2018, using ad-hoc sampling that combined a list of registered water shops from the Ministry of Health with exhaustive recruitment at filling areas and tanker parking areas (traffic circles where customers place orders). Coverage is believed to include all tankers active in Zarqa and East Amman. Content includes water sourcing, treatment, business characteristics (number of employees and tankers, income, expenditure), sales and shifts in customer habits or number, and external factors influencing their business including regional shifts in water supply. Questionnaires were prepared in English by SI, translated into Arabic, and tested prior to the survey for comprehension. The results were back-translated into English. Conducted in Arabic using Android Tablet. All questionnaires are provided as external resources.

Water Meter - questionnaires captured the identifying informatoin (meter number) for the utility meter, as well as the before-water turn readings for utility and ultrasonic meter and the after-water turn readings for utility and ultrasonic meter. Questionnaires also included pictures of both meters.

Non-Revenue Water - questionnaires captured the identifying information (plot number and meter number) for the utility meter, as well as the reading of the meter and a picture.

Miyahuna Maintenance and Billing Instruments - these data were provided by Miyahuna, and collected by Miyahuna teams. While specific details of the instruments used are not available, in general terms the maintenance database was created as recorded by Miyahuna call centers for customer complaints, and further updated based on the record of the maintenance team. The billing database was assembled based on records from Miyahuna's water meter readers, who record the reading each quarter. In cases where the meter could not be read, it is instead estimated from prior usage.

Household Survey - Urban areas of Zarqa and East Amman, with coding to the census block level

Agriculture Survey - Central and Northern Jorden Valley, Highlands, with coding to the locality level

Enterprise Survey - Urban areas of Zarqa and East Amman, with coding to the sub-area level

Vendor Survey - Tankers - Urban areas of Zarqa and East Amman, with coding to the district level

Vendor Survey - Water Shops - Urban areas of Zarqa and East Amman, with coding to the subdistrict level

National Aid Fund (NAF) Survey - households in urban areas of Zarqa, with coding to the census block level

Miyahuna billing and maintenance databases - Urban areas of Zarqa and Amman, with coding to the census block level

Water meter testing - Ubran areas of Zarqa, with coding to the DMA level

Non-revenue water testing - Two DMA's within Zarqa: RH4 and RL7