Global Program Review of the 2005 Independent Evaluation of the Development Gateway Foundation

Management response

The Foundation agrees with the major findings of the review:

• The Independent Evaluation (Muth/Gerlach 2005) was both organizationally and behaviorally independent; and
• During the two years since the evaluation was completed (and particularly during the tenure of the new CEO), the management of the Foundation has followed up on most of the evaluation’s specific recommendations and introduced a number of organizational, management and strategic changes including developing a monitoring and evaluation framework, and strengthening the governance structure by diversifying Board representation.

However, we would like to respond to a number of the review’s findings as the review process allows no substantive discussion of its analysis and recommendations:

• While the review is critical of the lack of outcomes-based performance indicators, we were disappointed by its failure to respond to our request for IEG’s expert guidance on what these might be and how they might be developed at reasonable cost. The review acknowledges that it is challenging to develop performance indicators in the ICT4D arena. The Foundation nevertheless remains committed to further developing its monitoring and evaluation framework in a way which is most meaningful for its stakeholders.
  
  
• A number of the detailed comments and conclusions on the effectiveness of the Foundation's programs by the reviewer appear to be based on a strict dichotomy between "pure public goods" and "private goods". As recognized in the guidelines for Global Program Reviews, the reality is a continuum of market imperfections that range from (almost) perfect private goods to (almost) pure public goods. For example, the review speaks of the potential "unfair advantage for dgMarket against competing suppliers of procurement software". This does not recognize that the "competitors" are (expensive) custom government procurement systems operated and paid for by governments. They are not bought off-the-shelf from private suppliers. We do not claim that dgMarket is a pure public good and the suggestion therefore that the nominal fees that are charged to recover some of its costs are inappropriate, is highly debatable.
  
  
• The Foundation developed software for a new e-procurement system for the World Bank in 2000 for no charge. This system was then intended to replace UN Development Business. In return, and as the UN decided to continue their on-line version, the World Bank agreed to make it mandatory to publish its tenders on both dgMarket and UN Development Business. As there is an automatic data feed from dgMarket to UN Development Business, there is no duplication of data entry. UN Development Business presently accounts for less than 4% of the money value of dgMarket’s coverage and has the same pricing as dgMarket.
  
  
• The review criticizes the dgCommunities user survey on the basis that the response rate was “too low for meaningful statistical analysis." The survey ultimately collected 1469 responses from a population of 36,000. The survey results therefore have a confidence interval of +/-2.5% at the 95% confidence level which is certainly statistically meaningful. The only potential methodological issue with the survey (as with any large scale web survey) is whether the respondents approximate a random sample of the population.

Download Global Program Review of the 2005 Independent Evaluation report [PDF]