IDENTIFYING IMPACT IN A TRANSNATIONAL PROJECT PROVIDING PARENT EDUCATION FOR FAMILIES LIVING WITH AUTISM IN SOUTH-EAST EUROPE
1Research & Innovation Funding Support, University of Northampton, UK
2Centre for Education and Research, University of Northampton, UK
Online first: 28-February-2020
1.1. Impact in research
Research impact is of crucial importance. The extent to which research has the potential for impact is a key factor in whether or not funding bodies support studies; and the extent to which it evidences impact is central to official national assessment processes such as the UK Research Excellence Framework (REF) (Martin, 2011; Smith, Ward & House, 2011) and Australia’s Excellence in Research for Australia (ERA) (Haslam & Koval, 2010). Despite this, the measurement of research impact remains imprecise.
Historically, research impact has been measured through journal and citation metrics, though this has long been identified as flawed (Amin & Mabe, 2004; Seglen, 1998). Research activity undertaken by academics should have broader impact than academic citation. The UK Economic and Social Research Council defines impact as ‘the demonstrable contribution that excellent research makes to society and the economy’ (ESRC, no date); and further identifies ‘academic impact’, the demonstrable contribution made with regard to shifting understanding and advancing theory and application; and ‘economic and societal impact’, the demonstrable contribution made to society, to the economy, and/or to individuals, organisations and nations.
Identifying non-academic impact of research can be problematic (Davies, Nutley & Walter, 2005). Engagement with research is one area of difficulty, as academic work is often viewed as inaccessible, complex or irrelevant (Parsons et al., 2013; Thomas, 2013). Moreover, research activity is not only undertaken by academics but requires input from research support personnel (Langley, 2012; Whitchurch, 2008). Recently there has been investment by research institutions to support the collection of impact evidence, for example impact officers (Bayley, Phipps, Batac & Stevens, 2017). The contribution of such personnel, as well as that of project managers and other support personnel, may valuably enhance impact and requires further research (Bramble, 2015; Fedorciow & Bayley, 2014).
It is acknowledged that impact methodologies often fail to demonstrate the effect of impact and evidence correctly when making judgements (Gartner, Cox & Jeffery, 2012) and the broadness of impact definitions and the variation in interpretation across disciplines may lead to imprecision and confusion (Kelly, Kent, McMahon, Taylor & Traynor, 2016). Therefore, the importance of developing effective assessment tools to identify impact in non-academic areas has been stressed (Bornmann, 2012). A number of such tools have recently been developed (Morrow, 2017; Thomas, 2013; Tsey et al., 2016); one such is the Erasmus+ Impact+ tool (British Council, Ecorys & CMEPIUS, 2016).
1.2. Erasmus+ Impact+ tool
Erasmus+ is a European Union programme, running from 2014 to 2020, funding activities within the fields of education, training, youth and sport. With an overall budget of €16.4 billion for this period (European Commission, 2018), the programme has been a major driver of social and educational inclusion (Cairns, 2017).
Proposals submitted for funding must demonstrate societal impact, addressing prescribed aims and priorities (European Commission, 2014). To help applicants identify how impact may be achieved, the programme’s UK and Slovenian national agencies developed the Impact+ tool (British Council, Ecorys & CMEPIUS, 2016) While the UK Research Excellent Framework defines impact as “effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (HEFCE, SFC, HEFCW & DfE, 2012), the Erasmus+ programme further refines the conceptualisation of impact, introducing an additional focus on internal impact within non-academic project partners and personnel.
The tool, launched by the European Commission in May 2016, considers impact within four discrete domains – systemic impact, impact to learners, impact to partner organisations and to project staff – and is intended to aid the development of outputs, outcomes and impact indicators.
1.3. The ESIPP (Equity and Social Inclusion through Positive Parenting) project
Equity and Social Inclusion through Positive Parenting (ESIPP) was a 3-year transnational project (2015-2018) which involved Belgian, British, Croatian, Cypriot and Macedonian partner organisations. The project’s focus was on the development, delivery and evaluation of parent education (PE) in autism within south east Europe, in areas where such support was previously unavailable or inaccessible to most families (Preece et al., 2017).
Activities undertaken within the project comprised:
- the identification and development of a PE curriculum and programme for families living with autism in Croatia, Cyprus and North Macedonia
- the development of local training teams within these countries
- providing PE to five cohorts of parents across the three countries (n=330)
- a mixed methods programme evaluation strategy using a range of quantitative and qualitative tools (pre-training, post-training and follow up parent questionnaires, parent interviews, trainer focus groups and reflective diaries)
- the development of a set of recommendations regarding PE for policy-makers.
Evaluation of survey and interview data with parents who attended PE identified that the workshops had a positive impact, improving their understanding of autism, and providing them with practical skills and strategies (Troshanska, Trajkovski, Jurtoski & Preece, 2018). As a result of the project, PE continues to be provided within the three project countries, as well as being introduced to other areas (e.g. Macedonian trainers took the materials to Kosovo in 2019).
ESIPP was identified as an exemplar case study in research undertaken for the European Commission regarding the use of the Impact+ tool (Williams & Bellemin, 2017). This tool was initially introduced to the project in September 2016 and was later used to focus discussion on impact at a transnational project meeting in March 2017. Though designed to help partnerships identify and agree a single model of impact, these discussions identified that partners held differing perceptions and attitudes regarding the project’s impact. To identify and understand these differences, a questionnaire was developed from the Impact+ tool by the lead partner organisation, the University of Northampton.
2.1 Data collection tool
A semi-structured questionnaire was constructed to capture project partners’ perceptions regarding the extent to which the project had addressed its identified aims and priorities, and the extent to which it had impacted (low, medium or high impact) across the four domains of impact. This questionnaire was developed in September 2017 and was piloted within the University and amendments subsequently made (see Appendix 1). For this reason, no data from this organisation are included.
Data were collected from partner organisation personnel attending the seventh ESIPP transnational project meeting (November 2017), 27 months into the 36-month project. All respondents were directly involved in the design, delivery and evaluation of the project. Thirteen respondents (81%) were female, and three (19%) male. Responses were gathered from all project partners except the University of Northampton:
- a trans-European non-governmental organisation (NGO) (n=1, 6%) based in Belgium
- a university (n=2, 12.5%) and school (n=2, 12.5%) from Croatia
- a university (n=3, 19%) and small training enterprise (n=3, 19%), from Cyprus
- a Macedonian NGO (n=2, 12.5%)
- a small training enterprise (n=2, 12.5%) and charity (n=1, 6%) from the United Kingdom (n=3, 19%).
Questionnaires were completed individually during the first morning session of the two-day meeting and took 30 minutes. Sixteen questionnaires were distributed and collected, a response rate of 100%.
2.3. Ethical issues
Ethical approval was obtained from the University of Northampton. All respondents provided written informed consent, and also consented to the data and findings being used in presentations and publications. All responses directly quoted within this article are anonymised.
Analysis was undertaken using basic descriptive statistics and qualitative thematic analysis (Boyatzis, 1998). Findings are presented firstly with regard to the priorities addressed by the strategic partnership, and then in relation to the four domains identified in the Impact+ tool.
3.1. Project priorities
ESIPP was established to develop, provide and evaluate PE in autism to Croatian, Cypriot and Macedonian parents, in three south east European countries where such support was non-existent or emergent. The project was aligned with three Erasmus+ priorities:
- Reducing disparities in learning outcomes affecting disadvantaged learners (in this case, Croatian, Cypriot and Macedonian children with autism)
- Enhancing the quality of early childhood education and care
- Strengthening quality through mobility and cross-border cooperation
Partners’ perceptions regarding the project’s impact concerning these priorities are identified in Table 1.
Partners’ perceptions of project impact on Erasmus+ priorities
Just under half of the partners felt a significant difference had been made for some disadvantaged learners.
Considering the children with autism being the disadvantaged learners, the project has worked well in empowering parents as educators and advocates for their own children – informing them of positive approaches that benefit the child’s autism-based learning style (Small enterprise, UK).
However, others felt impact was inevitably limited due to the size of the project –
The contents of the activities are valuable. Their impact could not be high due to the extent of the exposure of the participants to the training, due to what the project funded (Cypriot academic).
– and also pointed to the difficulties in identifying the extent of this impact due to the project’s focus.
Parents of disabled children are sometimes part of the disadvantaged groups. But we don’t know the impact on children themselves…. (There is) no way of identifying this (Cypriot academic).
Similar responses were gathered regarding the project’s impact on enhancing early childhood and care, with about half of respondents feeling that impact had been high –
Many parents that attended the training have young children, and they are using some of the strategies (that they learned) with their children (Croatian teacher).
– while others felt less sure, due to the focus on PE.
It certainly increases the quality because they gained a lot of tools, but it depends on how much and how consistently they will be applied by parents (Croatian teacher).
It was noted that partners from a range of countries felt the project did not address the topic of ‘education’ due to its focus on parents and the home.
The training was mainly addressed to parents. Educators did not attend the trainings in large numbers, therefore the project’s impact in this aspect is reduced (Cypriot academic).
The project is not education-based (UK trainer).
Given the long-established identification of the importance of consistency across settings and of a ‘twenty-four-hour curriculum’ to those on the autism spectrum (Forster, 1989; Jordan & Powell, 1995), such comments can seem surprising. However, such arguments are typically based upon scenarios where schools and educators have more knowledge and skills than parents, and where there is a need to share these with parents. Partners from the south-east European countries identified that ‘early childhood education is not very well developed’ and it was also suggested that training on how educational settings and parents can work together effectively is necessary.
Partners were more positive about the project’s third priority, perceiving high impact with regard to cross-border cooperation.
Having different partners interact and co-design the curriculum and training materials enhances quality of outputs/cross-border delivery of training (Cypriot academic).
This goal has been achieved with the transnational project meetings and stakeholder conferences, held in the different partner countries. We had the possibility to learn more about the other countries, their work and culture (Croatian teacher).
3.2. Domain 1: Systemic impact
Attitudes were mixed regarding systemic impact, with regard to the wider educational, political and social environment in project countries and beyond (see Table 2). It was telling that Cypriot partners identified the lowest systemic impact, whilst Macedonian partners – whose stakeholder conference has taken place shortly before, with significant media and political interest, identified high impact. This highlights how partners’ perceptions may be highly influenced by their own local experience. From a perspective of over a year after the project’s end, it is clear that systemic impact has occurred: however, this may be difficult to truly evaluate during a project’s lifetime.
Partners’ perceptions of the systemic impact of the project
Partners more readily identified potential barriers to impact within this domain. Key barriers identified included attitudinal factors (lack of awareness, ignorance, viewing autism as a ‘niche’ issue), economic factors (financial restrictions, competing demands) and local factors (continual changes of personnel in key governmental roles, lack of parental involvement within decision-making, cultural norms). Partners stressed the importance of dissemination activity – publishing journal articles and position papers – and working locally with parents to form self-advocacy and lobbying groups and identified actions to support this.
I will promote best practices and disseminate the project at a European level (Belgian NGO).
I will disseminate to policy-makers locally. Include the project as a reference in other work and link it to wider efforts towards inclusion (Cypriot academic).
I will talk with my director to take more steps to ask policy makers. Empower parents to ask policy makers (Croatian teacher).
3.3. Domain 2: Impact on learners
By contrast, partners were positive regarding impact on the ‘learners’, the parents who attended PE in the three countries. Partners made reference to both their interaction with parents during workshops and data collected from parents during the project evaluation. Data were collected from parents via questionnaires and interviews, and parental perspectives on their experience of training are reported elsewhere (Troshanska et al., 2018).
This has been a definite strength, as I have had family members personally share their new successes and increasing confidence with us (UK trainer).
Parents (from what they told us) gained a lot of useful information and strategies that they were able to apply to their homes and schools (Cypriot trainer).
Based on the data received, parents speak highly of the value (practical and social value) of trainings in helping them deal with the challenge of their everyday lives (Cypriot academic).
It was acknowledged that longer-term impact would depend on parental confidence and stamina, but there was a general perception that parents had been enabled to understand their children better and to develop new skills.
Parents have increased autism knowledge and strategies for coping. Some have referred to the programme as life-changing (Cypriot trainer).
Partners’ perceptions of the impact of the project on learners
As well as increasing parental understanding and teaching new skills – the central aims of the project –further impacts were identified. Parent-professional relationships were enhanced, with parents developing greater trust in professionals engaged in training and their skills. Parents reported a reduced sense of isolation through meeting and interacting with other families living with autism. It was felt this might help parents to ‘a realisation that they can have greater self-reliance and resilience’ (small enterprise, UK) enabling them to ‘become empowered to achieve better outcomes for their children’ (charity, UK).
3.4. Domain 3: Impact on partner organisations
Responses varied widely concerning the extent to which participation in ESIPP had impacted on partner organisations (see Table 4). This seems associated with the type of organisation to which partners belonged, and the extent to which engagement in such project activity was novel and innovative, or simply ‘business as usual’. Partners from higher education institutions reported low impact, as participation in transnational research and development projects is a norm.
This type of activity is expected of us (professors, staff on faculty) to participate (Croatian academic).
Though such activity was identified as an expectation, benefits were identified by academic partners.
It has encouraged us and the whole organisation/faculty/university is better developed (Croatian academic).
Participation in the project has helped our university expand its network of collaboration with EU partners through our involvement and has helped the university acquire knowledge and expertise through the partner involvement regarding autism and family needs/issues of inclusion (Cypriot academic).
Partners’ perceptions of project impact on Erasmus+ priorities
For other organisations, particularly schools, NGOs and small businesses, participation had a more significant impact, particularly with regard to increased visibility, the broadening of professional networks and reputational enhancement.
We, as an organisation, got more reputation through social media and from mouth to mouth within the country and abroad (Cypriot trainer).
Involvement in ESIPP had led some organisations to become more reflective concerning their work as autism practitioners.
The project has forced us to reflect on best practice, and the reasons why we think particular methods and approaches are best. It has reinvigorated our enthusiasm and reminded us that the autism knowledge base we possess is not universally available (UK trainer).
Practitioner partners involved in developing/delivering the PE programme identified that this activity ‘helped us to think beyond ‘local’ and to gain a wider perspective about autism issues and how they impact upon families’ (UK trainer).
Further positive changes identified by partners included the introduction of more structured approaches for managing data, improved inter-agency working and strengthened relationships with parent organisations. Teachers acting as trainers in the project had ‘started to use learned approaches in their everyday work’ in the classroom, thus broadening the impact of ESIPP to a wider range of children and families (Macedonian NGO).
Over 80% of respondents (n=13, 81%) identified that their organisation had developed new partnerships or services as a result of participation. Partners had developed Erasmus+ Key Action 1 mobility activities, enabling teachers from Croatia to visit the United Kingdom to observe practice in British schools. Partners had also been invited to join other projects or partnerships as a result of participation.
I was called to participate in a project that aims to develop support for parents of children with a range of different disabilities (Croatian academic).
3.5. Domain 4: Impact on project personnel
Participation in ESIPP was reported as having a high impact on participants themselves by almost 90% of respondents (n=14, 87.5%). Partners overwhelmingly identified participation as positive (see Table 5).
Partners’ perceptions of project impact on Erasmus+ priorities
All respondents felt that participation had impacted positively on their professional development, with nine respondents (56%) identifying high impact and seven (44%) identifying medium impact. Identification of medium impact was most frequent among academics, for whom participation in such projects was more typical. Areas of development identified by most partners included project management and organisational skills, improved English language skills and improvements in collaborative working. Participation provided opportunities for partners to reflect on different models of practice and working, ‘comparing my knowledge and professional style with others’ (Croatian academic).
Many partners identified specific areas of learning that participation had brought about for them.
I have learned new ways to approach different issues that can arise in working with families of children with ASD (Croatian academic).
I am now more flexible and more receptive on new ideas and opinions. I do try to be more cooperative with the people I work with (Cypriot trainer).
I have developed IT skills and am better at working online (Macedonian NGO).
I am more organised than before (Cypriot trainer).
Partners from all countries involved reported that they had increased their cultural awareness and sensitivity as a result of participation.
I have found out that there are more things that we have in common than things that separate us (Croatian academic).
Working within transnational team, I have learned about effective communication with persons with different cultural backgrounds (Croatian academic).
I am more tolerant of others’ ideas and the working practice of different individuals and cultures (UK trainer).
Finally, many respondents from across the whole range of participant organisations reported that they felt more confident and competent than before, due to their involvement in the project.
While I have perhaps not developed any new ones, I do feel an increased level of confidence overall thanks to the positive experiences of participation (UK trainer).
More competencies in work with parents (how and in what way to communicate, to present, and support parents (Croatian teacher).
It has been valuable being accepted as equal and valuable no matter what my academic title is (Croatian academic).
4.1. Identification of impact
The survey identified a range of impact across the four domains and also shed light on the partnership and its understanding of the parameters of the project. With regard to the four domains it is clear that the greatest consensus exists regarding impact on learners – the parents of children with autism who were trained. This was the project’s focus and a wealth of quantitative and qualitative data has been collected, analysed, shared and discussed by the partners. Personnel from all organisation types refer to ‘the data’ and ‘what parents told us’ to ground their perceptions in the evaluation process.
Systemic impact was more problematic to identify. This is unsurprising, as the project was ongoing at the time of the survey, and some project activities had not yet occurred or deliverables been developed. Furthermore, it is acknowledged that there can be a significant time lag between a project taking place and the full significance of its impact being identifiable (King’s College London and Digital Science, 2015; Morris, Wooding & Grant, 2011; Penfield, Baker, Scoble & Wykes, 2014). A problematic aspect of Erasmus+ projects (as with others) is the requirement in final reports to discuss impact across domains. Morton (2015) further argues that research utilization is both complex and interactive, with many factors affecting impact being outside the control of those undertaking the project, and it is suggested that while it may be possible at such a stage to identify local or short-term impact, an appropriate length of time must be allowed before wider and longer-term impacts can be meaningfully considered (Meagher, Lyall & Nutley, 2008).
With regard to partner organisations and partner staff themselves, the survey tool was able to identify impact, and also identified how such impact might be differently experienced dependent upon organisational and individual expectations and experience. Integrating personal and organisational learning and development is important not only to the individuals and organisations concerned (Bezhani, 2010; Fuller, Munro & Rainbird, 2004), but also with regard to the project as a whole, and we would argue that these examples of individual and organisational impact can be used to support the development of longer term impact indicators.
It is important to note the above findings reflect the Erasmus+ programme ethos of strengthening collaboration and team spirit, and partnerships between higher education and the external environment (European Commission, 2018). It is also important to emphasize that collaboration, partnership and engagement activity help the promotion and implementation of research policy and practice (Schnitzler, Davis, Ross & Harris, 2016), although how the partner organisations and partner staff use the findings of this study and any subsequent impact is yet to be determined.
There are clear limitations to this study. The project partnership had begun before the development of the Impact+ tool, and therefore it was not possible to integrate it into the project from the proposal development stage. Partners came to the project with a range of pre-existing knowledge, expectations and aspirations regarding impact, which were not formally explored until a year had gone by. Using the tool at the initial planning stage, and then throughout a project lifespan, could provide valuable insights into partner perspectives and support more effective impact planning and identification.
Overall, the researchers feel that the Erasmus+ Impact+ tool, and the questionnaire which we have adapted from it, have value and utility with regard to identifying impact to learners, partner organisations and project staff. Whilst acknowledging the limitations of these tools regarding the identification of wider systemic impact, we still feel that they have utility in this domain concerning the identification of potential barriers to impact, and to support sustainability planning activities. Most importantly, they enable project staff to develop their understanding of the skills and activities required to generate impact. They can support a broad and consistent conceptualisation of impact within project partnerships and teams, which we feel are equally relevant within European Union programmes and beyond.
Conflict of interests:
Authors declare no conflict of interests.
This work was supported by the European Commission under Erasmus+ grant number 2015-1-UK01-KA204-013397. This support does not constitute an endorsement of the contents which reflect the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
Amin, M., Mabe, M. (2004) Impact factors: use and abuse. International Journal of Environmental Science and Technology, 1(1), 1-6.
Bayley, J, Phipps, D, Batac, M., Stevens, E. (2017) Development of a framework for knowledge mobilisation and impact competencies. Evidence and Policy, online first.
Bezhani, I. (2010) Intellectual capital reporting at UK universities. Journal of Intellectual Capital, 11(2), 179-207.
Bornmann, L. (2012) Measuring the societal impact of research. European Molecular Biology Organization Reports, 13(8) 673-676.
Boyatzis, R.E. (1998) Transforming Qualitative Information: Thematic Analysis and Code Development. London: Sage.
Bramble, P. (2015) Business as usual: project management of a European project. In C. Devecchi (Ed) Volunteering as Empowerment for Success at School. Northampton: University of Northampton.
British Council, Ecorys & CMEPIUS (2016) Erasmus+ Impact+ Tool. Accessed 02/04/2018 at https://www.erasmusplus.org.uk/file/3624/download
Cairns, D. (2017) Researching social inclusion in student mobility: methodological strategies in studying the Erasmus+ programme. International Journal of Research and Method in Education, online first.
Davies, H., Nutley, S. Walter, I. (2005) Approaches to assessing the non-academic impact of social science research. Swindon: Economic and Social Research Council
Economic and Social Research Council (no date-a) What is impact? Accessed 09/04/2018 at: https://esrc.ukri.org/research/impact-toolkit/what-is-impact/
European Commission (2014) Erasmus+ Programme Guide. Version 2 (2015) 23/10/2014.
European Commission (2018) Erasmus+ Programme Guide. Version 2 (2018) 15/12/2017.
Fedorciow, L., Bayley, J. (2014) Strategies for the management and adoption of impact capture processes within research information management systems. Procedia Computer Science, 33, 25-32.
Forster, S. (1989) Towards a unified approach in a multi-disciplinary setting. In D.S. Baker, D. Baker & K. Bovair (Eds) Making the Special Schools Ordinary? London: Routledge.
Fuller, A., Munro, A., Rainbird, H. (2004) Workplace Learning in Context. Abingdon: Routledge.
Gartner, R., Cox, M., Jeffery, K. (2012) A CERIF-based schema for recording research impact. The Electronic Library, 31(4) 465-482.
Haslam, N., Koval, P. (2010) Possible research area bias in the Excellence in Research for Australia (ERA) draft journal rankings. Australian Journal of Psychology, 62(2), 112-114.
Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales & the Department for the Economy (2012) Assessment Framework and Guidance on Submissions.
Jordan, R., Powell, S. (1995) Understanding and Teaching Children with Autism. Chichester: Wiley.
Kelly, D., Kent, B., McMahon, A., Taylor, J., Traynor, M. (2016) Impact case studies submitted to REF2014: the hidden impact of nursing research. Journal of Research in Nursing 21(4), 256-268.
King’s College London & Digital Science (2015) The Nature, Scale and Beneficiaries of Research Impact: an initial analysis of the Research Excellence Framework (REF 2014) impact case studies. London: King’s College London.
Langley, D. (2012) Research management and administration. Perspectives: Policy and Practice in Higher Education. 16(3), pp.71–76.
Martin, B.R. (2011) The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster? Research Evaluation, 20(3) 247-254.
Meagher, L., Lyall, C., Nutley, S. (2008) Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Research Evaluation, 17(3), 163–173.
Morris, Z.S., Wooding, S., Grant, J. (2011) The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the Royal Society of Medicine, 104(12), 510-520.
Morrow, E. (2017) The Research Leader’s Impact Toolkit. London: Leadership Foundation for Higher Education.
Morton, S. (2015) Progressing research impact assessment: a ‘contributions’ approach. Research Evaluation, 24(4), 405–419.
Parsons, S., Charman. Y., Faulkner, R., Ragan, J., Wallace, S., Wittemeyer, K. (2013) Commentary – bridging the research and practice gap in autism: The importance of creating research partnerships with schools. Autism, 17(3) 268-280.
Penfield, T., Baker, M.J., Scoble, R., Wykes, M.C. (2014) Assessment, evaluations, and definitions of research impact: a review. Research Evaluation, 23(1), 21-32.
Preece, D., Symeou, L., Stošić, J., Troshanska, J., Mavrou, K., Theodorou, E., Frey Škrinjar, J. (2017) Accessing parental perspectives to inform the development of parent education in autism in south-eastern Europe. European Journal of Special Needs Education, 32(2), 252-269.
Schnitzler, K., Davies, N., Ross, F., Harris, R. (2016) Using Twitter™ to drive research impact: A discussion of strategies, opportunities and challenges. International Journal of Nursing Studies, 59, 15–26.
Seglen, P.O. (1998) Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica, 69(3), 224-229.
Smith, S., Ward, V., House, A. (2011) ‘Impact’ in the proposals for the UK’s Research Excellence Framework: Shifting the boundaries of academic autonomy. Research Policy, 40(10), 1369-1379.
Thomas, R. (2013) Research and scholarship with impact: a British perspective. Worldwide Hospitality and Tourism Themes, 5(3), 277 – 282.
Troshanska, J., Trajkovski, V., Jurtoski, F., Preece, D. (2018) The impact of ASD on Macedonian families and their experience of parent education. Journal of Special Education and Rehabilitation, 19(3-4), 127-138.
Tsey, K., Lawson, K., Kinchin, I., Bainbridge, R., McCalman, J., Watkin, F., Cadet-James, Y., Rossetto, A. (2016) Evaluating research impact: the development of a Research for Impact tool. Frontiers in Public Health, 4, 160.
Whitchurch, C. (2008) Shifting identities and blurring boundaries: The emergence of third space professionals in UK higher education. Higher Education Quarterly. 62(4), 377–396.
Williams, M., Bellemin, A. (2017) Impact+ Exercise User Research. London: Access Europe.
Copyright ©2020 Bramble, P., Preece, D. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)