'PUM Netherlands senior experts’. Evaluation positive?

PUM supports development of SME’s abroad. PUM has a network of 265 representatives in 70 countries around the world, connecting directly with entrepreneurs, business support organisations and partners locally. PUM works with 3,000 senior experts (volunteers) who share their knowledge on a one-on-one basis. Either through short-term and repetitive advisory missions at the work floor, or through online coaching activities.’ (https://www.pum.nl/how-we-work). 

I have been one of those volunteers

PUM just published the results of the evaluation of the PUM-programme 2012-2015. The evaluation is done by Erasmus University Rotterdam. (https://zoek.officielebekendmakingen.nl/blg-780351.pdf). The evaluation is partly based on the results of PRIME (Pioneering Real-Time Impact Monitoring and Evaluation in small and medium sized enterprises). See: www.primepartnership.nl.

The heading of the article in PUM-Magazine (autumn 2016) referring to this evaluation is (in Dutch) ‘PUM-evaluation positive’. This however seems to be only partly true and not for the most relevant part.

Reading the conclusions and the recommendations of the Erasmus report one would expect a more subtle heading. The researchers formulate numerous reservations on many evaluation questions. And luckily the short article in PUM magazine quotes several of those important sobering conclusions and recommendations. 
However, what should one conclude if  such a quote leaves out an important part? The details: In the next quote the underlined part is not included. ‘Missions to lower-income countries tend to involve more risks (the success rate is significantly lower), but if they are successful they generally result in significantly larger contributions to the programme’s and therefore of the ministry’s objectives’ (p. 58).
And what should one conclude if the only direct quote from the report in the article is: ‘Overall, PUM is considered as an efficient organisation. Beneficiaries are satisfied with the communication and speed of handling in the application stage’ (p. 7). 
Does this reflect the most relevant positive result of the evaluation? That would be rather meagre.

The heading also could have been more nuanced given the preliminary results of PRIME. PUM participates in the PRIME initiative to grow towards a better system for impact monitoring and evaluation.
Having worked in the field of Educational Management Information Systems at national level I am fully aware of the difficulties of designing, implementing and using such complex evaluation systems. PRIME is just a rather new initiative, so one can’t expect major results already now. The 2 pages summary of PRIME’s work regarding PUM reads: ‘PRIME needs more years of data to verify if these business practices contribute to a better business performance’ (p. 2). That is understandable.
So at this moment PRIME can’t conclude that PUM is contributing to better business practices that in turn contribute to better business performance. It is still an assumption.

But the PRIME summary continues: 'The wider impact literature is, however, supportive to this assumption. A recent systematic review of 40 impact studies shows that business training to SMEs does improve their revenue and profits, their ability to create jobs, labour productivity and their ability to invest'. (Piza e.a., The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low- and Middle-Income Countries: A Systematic Review, 2016).

Again it is rather sobering to read the Piza report from the Campbell Collaboration. It starts of well:
‘On average, business support to SMEs improves their performance, their ability to create jobs, their labour productivity and their ability to invest. The effects on innovation are unclear.
Matching grants, technical assistance and tax simplification programmes improve firms’ performance and job creation; with technical assistance also improving labour productivity. Export promotion and innovation programmes positively affect exports and innovation, but there is no evidence that they improve performance or job creation. 

But he next sentence warns:
However, the effects of the programmes studied are not very large. Most studies do not include the required data to assess if the programmes are cost effective.’ (p. 7)

And reflecting on the results of the research the Piza report states:
‘Overall SME support has a positive impact on various measures of firm performance, but with some caveats. Results for all the interventions studied could not be provided due to a lack of evidence. And the evidence available was mainly about programmes in Latin American countries. There is a likelihood of bias in many studies.' (p. 7)

So, PRIME relies on Piza but but the evidence that is sought is not presented there.

With such a conclusion about PRIME and with the reservations of the Erasmus researchers I leave it to you to formulate a better heading for the article in PUMmagazine.


Go to top