A Composite Index for the Benchmarking of eHealth Deployment in European Acute Hospitals Distilling reality into a manageable form for evidence-based policy

In a previous post entitled Benchmarking HIT Adoption in European Healthcare Organisations several challenges, including transparency, were mentioned. To tackle of those challenges, during the past few months I had the pleasure to collaborate with my colleague Cristiano Codagnone in the development of JRC Scientific and Technical Report entitled “A Composite Index for the Benchmarking of eHealth Deployment in European Acute Hospitals Distilling reality into a manageable form for evidence-based policy” published May 2011 .

Compared to other areas of the Information Society, where benchmarking has been conducted more systematically for longer (i.e. eGovernment), it is evident that benchmarking of eHealth deployment is lagging behind.

In this context, the results of the eHealth Benchmarking, Phase III survey, carried out by Deloitte and IPSO on behalf of Unit C4 of DG INFSO, with the rich information provided on about 1,000 European acute hospitals, could be a strategically important tool to close this gap. As we show in more detail later, this survey sheds light on key issues such as hospitals’ deployment of ICT infrastructure, applications, and much more.

The reasons why benchmarking of eHealth deployment is lagging behind are structurally related to the multi-dimensional complexities of this field, to the relatively greater difficulty/costs of getting the data (i.e. data cannot come from web-based measurement, as it can for eGovernment benchmarking), and especially to the challenges of making sense of the data.

This report uses multivariate statistical methods to analyse with a selective but deep vertical focus the results of the above-mentioned survey. The objectives of this exercise are two-fold:

a) to make sense of the results by constructing a composite index;
b) to extract key policy messages and new directions for future research.

The main objective is the elaboration of a composite index of eHealth deployment with a view to proposing a roadmap towards systematised and replicable benchmarking. In addition, we also explore the possible link between benchmarking and eHealth impact.
Therefore, our focus is much more selective but deeper than the broader descriptive analysis produced by Deloitte and Ipsos. In addition, we do not simply conduct multivariate statistical analysis but we put this into a conceptual and theoretical perspective and we follow it with a discussion of the results and with a set of policy and research recommendations.

This first introductory section is followed by four more. Section 2 provides the general conceptual and theoretical framework for benchmarking within an international policy perspective. Section 3 presents the data and the methodology used. In Section 4, we present and comment on the results of our multivariate statistical analysis. Finally, in Section 5 we discuss these results and extract recommendations for future research and policy making.

The Composite Index

The Hospital eHealth Deployment CI has been developed following a totally transparent multistage approach, which is graphically rendered in the figure below:

Countries with more intensive (per capita) healthcare spending in ICT score higher in our hospitals eHealth Deployment CI and it seems now perfectly sound that Italy, France and Germany have lower than expected CI in view of the fact that their ICT expenditure is considerably less intensive than in countries such as for instance Denmark, Sweden, and Norway. The data used are too aggregate and we do not dare going further than simply pointing out a mere statistical association. Yet, at least the direction is comforting: if it was negative (high rank in CI associate with low level of spending intensity) than we might have had a problem.

We replicated the operation done with ICT expenditure in healthcare with the following supply side indicators: “Hospital beds – Per 100,000 of population”; “Practising physicians – Per 100,000 of population”; “Number of Computer tomography scanners per 100,000”.

Again we stress that our aim was explorative and we looked for mere trends and statistical associations, with no claim to demonstrated significant statistical correlations and even less so infer causal relation. Yet, all of the trends illustrated in the following figures are comforting and not counterintuitive with respect to what one would expect as a result of wide introduction of eHealth on the above three supply side indicators: a) it would be counterintuitive and challenging to find the our CI is higher in countries with the highest number of hospital beds; b) it would be counter-intuitive and challenging to find the our CI is higher in countries with the lowest number of practicing physicians; c) it would be counter-intuitive and challenging to find the our CI is higher in countries with the highest number of computer tomography scanners. The trends in the figures do not support such instances. Naturally, we do not claim that having a higher CI enable to use fewer beds, to support more physicians, and to substitute scanners, for a much more in depth and granular analysis would be needed to substantiate this hypothesis. We simply observe that at least the direction of the trend is in line with what one may expect from relatively higher deployment of eHealth in hospitals.

Despite very relevant comparability problems, we can risk concluding that the results of the eHealth Benchmarking Phase III survey show that progress has been made in Europe with respect to the levels of eHealth deployment registered in previous, less systematic and extensive data gathering activities such as Business Watch and Hine. For instance, the penetration of Electronic Patient Records (EPRs) has increased from the 34% reported for 2006 by Business Watch to the current 81%. This 81% penetration of EPRs puts
Europe way ahead of Japan and US, where only between 10% and 15% of hospitals have introduced them. However, there are also several indications of areas in need of policy action, of which we emphasise the following four:

1) The CI shows large scope for improvement. The average EU27 CI stands at 0.347, whereas that of top scoring Sweden is just slightly above 0.5. This means that there is still room for general improvement.

2) Wide variation across countries. In particular, the lowest deployment measured by our CI is concentrated mostly among the new Member States and candidate countries. Of the bottom 13 countries, 12 are from this group – Greece is the exception. The only new Member State that scores above the EU27 average is Estonia, confirming its excellence in the domain of ICT. This calls for awareness raising policies and possibly financial support targeting this group of countries.

3) The summary indexes of the four dimensions identify areas to be prioritised. Whereas infrastructure deployment is quite high in most countries, electronic exchange of information lags behind fairly generally (across countries). It is important to close this gap, since these exchanges constitute one of the pillars of the vision and promises of ICT-supported integrated personal health services. These services are the key to producing better health outcomes while pursuing system sustainability and they must be developed around a seamless view of the user, for which exchange of information and timely clinical decisions are crucial. Yet, our analysis shows that electronic exchanges are still limited among the potential interacting players. Furthermore, cross-border exchanges are extremely limited, a gap that from the perspective of EU policy should be quickly addressed.

4) Predominant intramural orientation. From both simple descriptive statistics and from our multivariate statistical analysis, it emerges clearly that the deployment of eHealth in hospitals has been predominantly focussed on intramural needs and applications. For instance, levels of deployment for Personal Health Records and home-based Telemonitoring are very low. We need to stress that if the objectives and targets of the upcoming European Innovation Partnership on Active and Healthy Ageing are to be realised, much more progress will be needed in terms of both electronic exchange of information and user-oriented applications and services, such
as PHR and Telemonitoring.