The value and limitations of core tech transfer metrics to measure impact
Executive summary
This post reflects on the value and limitations of technology transfer metrics to measure social and economic impact. Considering that the core metrics are proxies for impact, the post offers recommendations on how to use them most fruitfully. It recognises that metrics are necessary for funders and university administrations to understand the social and economic impact of technology transfer activities. But it also highlights the limitations of the core metrics—number of disclosures, number of patents, number of licences, number of spin-outs and licensing income—in measuring an impact that is several steps and years away from the origination of these metrics. Drawing on a series of positive US, UK and EU developments involving metrics, the post provides recommendations to be considered when collecting, analysing and interpreting metrics. Crucially, to strengthen the understanding of the value of technology transfer, it calls for the collection of further metrics along the pathway from research outputs to impact by involving the many actors invested in the pathway.
Please feel free to leave any comments below and join the debate!
Photo by William Warby on Unsplash
Introduction
Over the last two years, the international technology transfer (TT) sector has contributed to the launch of a lively debate around key publications on metrics. These have included the ‘A la carte Menu’ produced by the US National Institute of Standards and Technology (NIST) in 2020 as part of the various exercises resulting from its Green Paper; Alison Campbell et al’s 2020 report ‘Knowledge Transfer Metrics – Towards a European-wide set of harmonised indicators’ commissioned by the European Commission’s Joint Research Centre, and the results of the first iteration of the Knowledge Exchange Framework published earlier in 2021 by the UK’s Research England. The size and complexity of these publications demonstrate that measuring TT activities is no easy task and that much thought needs to be given to any such exercise.
This post adds to the debate by recognising the fundamental value of measuring TT activities but also pointing to the many limitations of the core metrics collected by technology transfer offices (TTOs) to measure different outcomes, most controversially impact. The metrics discussed in this post are the core metrics collected by TTOs—number of disclosures, number of patents, number of licences, number of spin-outs and licensing income—and so the focus of this post will be on licences and start-ups. However, most of the recommendations below apply to the measurement of knowledge exchange activities more widely.
Technology transfer has grown significantly over the last quarter of a century, becoming adept at supporting the development of companies, either through licensing or creating innovative start-ups, and helping economies to flourish. The UK government counts on this expertise in its Innovation Strategy, and so does the US Senate when it passed the recent US Innovation and Competition Act (USICA) Bill supporting, inter alia, the public funding of university TT. The centrality that TT is gaining in public efforts to foster a sustainable post-pandemic economic recovery will, in turn, demand greater attention to the performance of university TTOs and to the social and economic impact that TTOs generate.
As this post will argue, core metrics collected by TTOs—number of disclosures, number of patents, number of licences, number of spin-outs and licensing income—are useful at answering some questions on performance, but cannot alone answer all questions. Many more metrics, both quantitative and qualitative, need to be added to the TT toolkit to answer questions about social and economic impact, whilst keeping an eye on the burden of collection of these additional metrics. But quantity is not the only issue: defining the metrics used appropriately and interpreting them consistently so that they are comparable; offering context so that differences are understood, and using reliable sources, are among the many issues that challenge measurement exercises. This post aims to address these and also remind users of these exercises of the risks of focusing too much on the measurement exercises alone, rather than on actual outcomes.
The post concludes with some recommendations on producing and reading measurement exercises. Crucially, to strengthen the understanding of the value of TT, it calls for the development of further metrics along the pathway from academic inventions to the generation of social and economic impacts whilst making sure to spread the burden of collection amongst the many actors involved in the pathway. This will be of key importance in providing a strong evidence-base to support the UK’s Innovation Strategy and other international policy efforts drawing on TT’s important offer.
1. What are metrics for?
When collecting and using metrics, it is important to be clear about what questions we are seeking to answer. For example, is it to measure the activity of a TTO or its internal performance against the expectation of university management? Is it to benchmark a TTO against others to identify relative performance? Is it to measure the impact of a TTO on a local ecosystem or on the wider economy? Is it to measure a local ecosystem or the effect of the surrounding ecosystem on the TTO’s activities? Or is it all of these? They all may require different sets of metrics that may or may not be under the control of TTOs.
In a confidential study commissioned in 2017 by the University of Cambridge’s research commercialisation office, Cambridge Enterprise, for its five-year review, the authors, Tomas Coates Ulrichsen and Tony Raven, were asked to assess whether the TTO’s numbers of disclosures, patents, licences, spin-outs, and licensing income, were in line with common practice. While specific details of the study remain confidential, overall findings were indicative of common practice amongst 18 high-performing US, UK and other European TTOs at the time of the study.
The authors used the database of the Association of University Technology Managers (AUTM) because of its well-established information on US universities and its definition of licences (also used by Cambridge Enterprise). In addition, the authors worked with the individual universities to check that the data was correct and the definitions were consistently understood.
The authors showed that the number of disclosures per $100m of research income of all the TTOs surveyed lay somewhere between 40 and 60 disclosures. In the same study, the authors found that patents filed per $100m of research income tended to be higher in the US than in the UK and other European countries. The authors brought forward the hypothesis that the differences might be due to the Bayh-Dole Act, which mandates the filing of disclosures from federally funded research, and so recommended inspecting this difference further.
But patents alone are poor proxies of impact; more telling are licences of value (see, for instance, Campbell et al 2020: 22). The authors found that the number of licences of financial value (as defined by AUTM) per $100m of research income varied widely between the TTOs surveyed, but it was consistently below 18, with a median of 8.5 licences per $100m of research income. Another related insight showed the degree of conversion of disclosures to licences of value. Here the authors found that 10 to 77 per cent of the disclosures taken forward by this group led to licences of value, with the majority of TTOs sampled lying somewhere between 10 and 33 per cent.
These are interesting findings that make full use of the core metrics collected by TTOs, but it is worth noting that the findings relate only to the participating TTOs’ performance, and this only in relative terms. This is important: the benchmarking exercise does not tell a TTO whether it is performing well in absolute terms, but whether its performance is comparable to that of its peers. It also gives an indication to university managements and governments of what can typically be achieved for a fixed amount of research income. Conversely, if the study determines that a TTO’s performance lies outside of the common ranges, more extensive research would be required to identify whether this is because the TTO is unproductive or, in fact, innovative.
While the study offers valuable insight on the relative performance of TTOs, much of the debates surrounding TT metrics revolves around understanding the social and economic impact of TT.
2. What is impact?
In 2006, the UK Warry Report (on the economic impact of research councils) provided a broad definition of economic impact that is still widely cited:
An action or activity has an economic impact when it affects the welfare of consumers, the profits of firms and/or the revenue of government. Economic impacts range from those that are readily quantifiable, in terms of greater wealth, cheaper prices and more revenue, to those less easily quantifiable, such as effects on the environment, public health and quality of life. (Warry, 2006)
Greater efforts for research accountability began in the early 1990s in the UK. Since then, the government has led the international trend to quantify research excellence and impact by creating organisations and mechanisms to systematically collect and process quantifiable economic impacts. Technology transfer is one of the many forms in which universities engage with communities and produce impact and is measured with the core metrics outlined above.
The above-cited 2020 report on metrics by Alison Campbell and collaborators, commissioned by the European Commission’s Joint Research Centre, rationalised the knowledge flow from research outputs to impact in this convenient summary box. It clearly sets out the complexity of the pathway between a research output and its impact, as would, for instance, be job creation. The report illustrates in a simplified manner the important translational work carried out through the KT channels (after ‘knowledge transfer’, the term most commonly used in continental Europe for what is known in the UK as ‘knowledge exchange’). The work of TT professionals is part of it, and may include consultancy, licensing and company creation.
Knowledge tranfer: from research to impact (Campbell et al 2020: 6)
3. Core technology transfer metrics as proxies for impact
In order to fund the activities of a TTO, universities and governments need to understand the social and economic impact of those activities. The key metrics employed around the world for this are number of disclosures, number of patents, number of licences, number of spin-outs and licensing income, together with some others related to and supporting these. While they are relatively straightforward to collect, they are imperfect measures of impact.
Consider the long and serendipitous pathway from research to company formation and impact. When an inventor comes through the door of a TTO to disclose an invention, the TTO must decide whether it will take the invention forward and, if so, whether it will seek to patent the invention or whether it needs to be protected in any other way.
Say that the TTO decides to commercialise the invention, that the invention is patentable, and that therefore the TTO decides to patent the invention. This is a costly endeavour that can absorb tens of thousands of pounds and a few years depending on the countries where it is filed and whether the filing or patent itself is contested. In the meantime, the inventors and TTO market the invention in order to attract interest from a potential licensee capable of developing the technology and take it to market. In the many cases in which the inventors do not find a suitable licensee, they may consider starting up a company, usually with the help of the TTO.
Here, another number of stages needs to be followed: the inventors will need to demonstrate proof of principle, create a business plan, develop a prototype, build a market for the technology and secure financing. Once the spin-out secures funding and leaves the university environment, the spin-out's success will depend on the strength of the team and business plan, the potential of the technology, the readiness of the market, the economic environment and other factors that may not be under the control of the company's managers, such as the pandemic. Of the surviving spin-outs, a fraction will lead to an exit either through an initial public offering or an acquisition.
So, let's return to measuring impact and the five core metrics: disclosures, patents, licences, spin-outs and licensing income. Considering the serendipitous and multi-stage pathway from disclosure to licensing or exit, these measures represent proxies, imperfect variables that stand in and approximate the amount of the impact. The first three represent early stages in a longer trajectory that may take anything between 3 to 15 years on average. The last one gives a sense of how much income a TTO receives over the lifetime of all the agreements and exits taken together. However, an extraordinary growth spurt or a highly successful exit can distort the picture significantly: can this success be tracked back to the TTO? And what does it say about the TTO’s current management, 15 years on?
Note also that none of the core metrics is in itself directly indicative of the impact created further down the line, as would be, for instance, jobs created, services sold or lives improved, but they are the ones that are under the control of the TTO and therefore easiest to collect, assuming the TTO has the right support to collect this data. Yet together, they can help to construct a better picture of a TTOs' impact and of the overall TTO landscape.
Are there better proxies for measuring impact? Yes, there might be, but at the cost of TTOs documenting every decision and asking their licensees and end-users to do the same over the many years in which the relevant companies may be active. This exercise would be excessively costly for TTOs, and anything in between might be too, as TTOs already have enough on their plate delivering their core activities and collecting the above metrics. That said, some additional metrics are collected and tracked by TTOs and other actors involved in the pathway. They provide additional detail for deeper investigations into TTO activity in specific clusters or sections of the pathway but, unfortunately, they are not collected in any consistent way in any part of the world.
This does not mean that TTOs do not know what types of impact academic inventions set out to achieve. The most successful impact stories are reported best in narrative format, though, such as it is done by the UK Research Excellence Framework (REF) or the Better World Project published by AUTM, which document the impact that individual pieces of research have on groups of people, their well-being and their environment. They may be such amazing impact stories as are a vaccine protecting against COVID-19, a new generation of cancer treatments, or a plant-based biodegradable replacement for single-use plastics.
These impact stories are illustrative of the potential of academic inventions to positively transform people’s lives, but funders and university administrations also need to keep track of the relative performance of TT activities to justify their investment. We need proxies to do this, as tracking the complexity of the tech transfer pathway would otherwise be excessively costly both in time and resources[1]. There is therefore also a common understanding that the measurement of TTO impact via proxies is by necessity only an approximate. How approximate depends on a series of caveats, of which this post outlines a few.
4. Defining metrics
It should be simple enough to collect the number of disclosures, patents, licences, spin-outs and licensing income–if only we knew exactly what each of them meant.
To take an example, the UK Higher Education Statistics Authority (HESA) defines a disclosure as: ‘the point at which academic staff disclose their idea through a formal process with the prospect of seeking protection’. As narrow as this definition appears at first glance, it poses at least two key questions: should TTOs report on every disclosure that academic inventors think is worthy of protection? Or should TTOs in fact report on the disclosures that TT professionals identify as worthy of protection? Much of the expertise of TT professionals involves identifying the value proposition of a new idea which leads to advising many researchers to publish or continue developing their ideas, as distinct from taking the ideas forward to commercialisation. A TT professional confided that ideas taken forward represented around 50 per cent in the physical sciences and 10 per cent in the life sciences, so using different interpretations may have significantly different outcomes.
Another example is the definition of start-ups by AUTM. For AUTM, a start-up is a company dependent on licensing an institution's technology for its formation. At a recent event held by AUTM, US TTO Directors met to discuss what start-up metrics said about their profession. It became clear that directors stretched this definition to report on different things. One speaker collected metrics and reported on every single start-up because ‘I’m not going to tell an entrepreneur that their start-up is not a start-up’. Another speaker collected metrics on every start-up but reported only on start-ups that had raised venture capital investment. Their reasoning was that if someone asked them to visit 20 of the 25 companies created in their TTO, they would struggle to show them more than five. They therefore only reported on these five. These are two different ways of interpreting the definition of a start-up that result in vastly different outcomes[2].
These examples show that the definitions are open to interpretation and that even when they are arguably specific, not everyone understands or uses the definitions in the same way. This means that often the metrics produced by each TTO are not necessarily directly comparable. This problem deepens when trying to compare metrics across different regimes, for example, metrics collected by the UK’s HESA with that collected by the US-based AUTM.
5. Context is everything
When reading through the metrics collected by each university by bodies such as HESA, AUTM and those of other countries, it helps to know the context of each university. When comparing the metrics on spin-outs of the two TTOs cited above, for example, we now know that it is likely that even if both TTOs produce the same number of spin-outs, their reported metrics are likely to be very different.
A well-known example, similar to the one outlined above regarding the definition of start-ups, is that of start-ups from arts colleges. Art and music graduates and staff often create limited companies to offer their products and services. These companies, which may have been formed based on intellectual property created within the institution, are not always scalable and are unlikely to attract venture capital investment. The large number of the companies compete directly on paper with that of research-intensive universities, even though the outcome in terms of economic impact is likely to be radically different.
The context of each university is therefore key. This has been recognised by the UK’s Knowledge Exchange Framework, which used a sophisticated framework to group universities with similar characteristics into clusters and, in addition, required three brief narrative statements of a university’s context and particular policies as part of its first submission in 2020. It remains to be debated whether this information is sufficient, but that it links hard numbers with context is to be celebrated and provides an alternative to other regimes, where TT professionals depend on interaction with other TT professionals and other forms of insider knowledge to read the resulting metrics effectively.
6. Can we trust the metrics
Considering all of the above, it becomes clear why league tables published around the world vary so widely in their results. This is good for TTOs in that they can choose the leagues that suit them most, but ultimately, this is an indication of their limitations.
Yet there are some indicators that make results more or less trustworthy, and this comes typically down to the rigour with which the metrics were collected and analysed. Regarding the collection of metrics, is it clearly indicated how they were collected and does the methodology for collection add up? If metrics were not collected first-hand, is the source a reputable one? If the source is compromised, as it often is for all the reasons above, have the authors sought to amend this by, for instance, talking to the TTOs in question to clean up their metrics? Regarding the analysis, is the methodology clearly indicated and does it stack up? If the methodology has limitations, are they flagged up?
These are all important questions to ask when reading league tables and studies that rank TTOs by performance, impact, investability and so on, and that will help to separate trustworthy studies from those that are less so.
7. False incentives - unintended consequences
A common concern around the collection of metrics to measure impact is encapsulated in the phrase (attributed to former Director of MIT TLO, Lita Nelsen): WYMIWYG (what you measure is what you get), a paraphrase of WYSIWYG (what you see is what you get). A TT version of Goodhart’s Law, the phrase makes the point that the metrics collected in order to measure impact, become the end in themselves. In other words, it makes the point that TTOs may end up collecting patents or spin-outs in order to improve their metrics, but not to generate impact. It may also underpin poor interpretation of definitions such as that of ‘disclosures’ or ‘start-ups’ exemplified above in the way that offers the largest numbers.
A study published in Nature Biotechnology built on that idea, accusing some universities of churning out spin-outs without a real potential for growth, ‘the living dead’. The authors proposed that to counter this situation, universities and governments should align rewards with desired outcomes (i.e. social and economic impact). This would involve collection of a broader set of metrics such as number of employees, total wages, other expenditures, VC raised and any liquidity events. But these metrics are not under the control of TTOs and are therefore much harder to collect. This is because once spin-outs leave the university environment, it is difficult to follow up on their activities, which, over time, may take different directions.
This post sides with the above authors in recommending governments to make a concerted policy-effort to collect metrics along the entire pathway from research outputs to social and economic impact by mandating the collection of metrics also to entrepreneurs and investors. In doing so, governments need to make sure that the burden of collection is distributed fairly across the pathway and a fair balance is struck between the quality of the metrics and the burden of collection at each point. Some relevant information is already publicly available through company accounts and tax filings, but it is unevenly distributed. Some key information is also missing: for instance, how can acquisitions be accounted for?
Mindful of the burden of collection imposed on federal labs, the US National Institute for Standards and Technology (NIST) has taken an interesting approach: US federal laboratories are obliged by law to submit annual performance reports to the Office of Management and Budget (OMB) and to NIST. The key expected metrics are grouped under ‘invention disclosures and patents’, ‘licences’, ‘income from licensing’, ‘collaborative research and development agreements and other collaborative agreements’ and ‘other performance metrics deemed important by the agency’. For the last group, in 2020 NIST collated an ‘A la Carte Menu’ listing a number of metrics that are currently reported by at least one agency and is intended to be informative for agencies as they determine what additional metrics they may like to report on. Over time, NIST can then assess what metrics are used and preferred by federal labs and suggest a new list without imposing these a priori.
8. Collection of metrics across the pathway to impact
Governments are increasingly relying on universities to generate social and economic impact over and beyond that provided through research and teaching through their knowledge exchange activities, which include technology transfer. In order to support robust policy-making, governments will need to gain insight into the emergence and development pathways of spin-outs to understand the factors shaping these pathways and maximising impact. This will require a more comprehensive set of metrics to give a clear indication on the emergence and development of spin-outs along a number of key dimensions (e.g. investment, technology, workforce, market traction/penetration, cost-saving, welfare etc.).
Yet TTOs are not in a position to collect these metrics. As Campbell et al observed in their report, the collection of such metrics is beyond the remit of TTOs (2020: 23). Instead, governments should seek to distribute the burden of collection across the pathway to impact by asking entrepreneurs and investors to provide adequate metrics. Metrics should also be held within a single, easily accessible national registry so that policy officials, researchers and others are able to undertake impact-related analyses for the public good. This would be consistent with the new UK National Data Strategy and could build on efforts such as that by the UK Office for Life Sciences in developing a comprehensive database of all companies known to be involved in the UK life sciences industry.
Recommendations
In summary, universities and governments need TT metrics in order to assess the social and economic impact of TT activities. The core TT metrics employed for this—disclosures, patents, licences, spin-outs and licensing income—go some way in measuring the activities of TTOs and benchmarking them against their peers, but are imperfect proxies for measuring impact. Considering this, the following list outlines some recommendations on how to use metrics more fruitfully when collecting, analysing and interpreting metrics. The recommendations follow from the points above and are organised by section for reference.
Crucially, in order to strengthen the understanding of the value of technology transfer, this post calls for the collection of further metrics along the pathway from research outputs to impact by involving the many actors invested in the pathway.
Different questions require different sets of metrics: core technology transfer metrics are best used to measure relative technology transfer performance, and should be used only as the basis for further enquiry.
3. Core technology transfer metrics as proxies for impact
Remember that core metrics do not directly measure impact, they are proxies for impact that balance the quality of the metrics with the burden of collection.
Make sure definitions are as specific as possible and account for potential divergences leading from different interpretations.
Seek and/or provide information about the context in which the metrics should be read.
Ensure that metrics are trustworthy; this might involve double-checking with the sources, especially when the metrics are not collected first-hand.
Ensure the methodology is clearly indicated and stacks up.
Seek and flag up any limitations that the methodology may have.
7. False incentives – unintended consequences
To the extent that it is possible, universities and governments should seek to align rewards with desired outcomes (i.e. social and economic impact).
8. Collection of metrics across the pathway to impact
To gain insight into the value of technology transfer, collect further metrics along the pathway from research outputs to impact from the many actors invested in the pathway.
Seek a fair balance between the quality of the metrics and the burden of collection.
Notes
[1] Rare studies of the life cycle of an invention and its impact do exist: for example, the once in a lifetime report commissioned by the US National Institute for Standards and Technology of the impact of GPS, an invention linked to US federal lab research.
[2] To confuse matters further, in the context of the UK’s HESA, the closest definition would be that of a ‘spin-off’. ‘Start-ups’ in this context are companies set up by the university but not based on university IP.