EMBO is a founding member of the San Francisco Declaration on Research Assessment (DORA), a global initiative to advance practical approaches to research and researcher assessment across all scholarly disciplines.

Since the Declaration’s publication in 2013, DORA has transformed from a call to action into an active initiative. EMBO actively supports DORA.

Bernd Pulverer, Head of EMBO Scientific Publishing, serves on DORA’s Steering Committee.

To celebrate DORA’s 10th anniversary EMBO and EMBL have organized a joint panel session: Exploring best practices in research assessment: EMBO, EMBL and DORA, on 12 May 2023.

EMBO has reinforced its commitment to improving research assessment by signing the Agreement to Reform Research Assessment, drawn up by the Coalition on Advancing Research Assessment (CoARA) in January 2023. EMBO has formed an internal working group to review its research assessment processes and ensure they are in line with the commitments of the agreement.

Examples of EMBO policies and initiatives for its funding schemes that align with the DORA aims and recommendations:

1. DORA general recommendation: Do not use journal-based metrics as a surrogate measure of the quality of individual research articles, to assess individual researchers or funding decisions:

• The EMBO guidelines for reviewers instruct them not to rely on journal impact factors (JIF) or other publication-based metrics when evaluating applications.
• The EMBO guidelines for applicants instruct them not to indicate JIF or any publication-based metrics, and the EMBO office deletes them from applications if they still appear.
• Another measure implemented to focus reviewers’ attention on the content of papers rather than on metrics is asking applicants to provide details of their best two to three papers, including a summary and a short explanation of why they chose those papers (two papers for the Installation Grants  and three for Young Investigators and Postdoctoral Fellowships), rather than just ask for a list of all publications. We are working on improving this further.

2. DORA recommendation 2 (for funders): Be explicit about the criteria used:

• Guidelines for applicants and reviewers are available on the EMBO website and provide detailed descriptions of the criteria used for each of its funding schemes: https://www.embo.org/funding/fellowships-grants-and-career-support/

3. DORA recommendation 3: Consider the value and impact of all research outputs:

• EMBO accepts preprints posted on preprint servers alongside articles published in peer reviewed journals.
• Since 2022, for the Postdoctoral Fellowships, a first author preprint with in-depth peer reviews publicly available from a trusted independent preprint peer review platform is also sufficient for eligibility. Whether peer reviewed preprints can be considered also for full applications for some of the funding schemes is under consideration.
• EMBO has analysed options to reward peer review activity in research assessment and together with other institutions is planning to develop pilots to test including peer review in criteria for assessment.
• EMBO is discussing introducing broader criteria to distinguish candidates in the so called “grey zone”.

4. DORA on openness: “Open scholarship allows access to works through various platforms, not only journals, and so facilitates evaluation of works on its own merit.”, from https://sfdora.org/2020/08/18/the-intersections-between-dora-open-scholarship-and-equity/:

• EMBO measures toward openness: EMBO is accepting preprints in applications, which makes it possible for researchers to share the results of their research openly (and faster).
• Open Science: EMBO is currently discussing which Open Science practices can be rewarded in its funding schemes, and how to evaluate their quality.

5. DORA has discussed the use of narrative CVs as a qualitative method for research assessment, for their potential to support a research culture that emphasizes meaningful research achievements over the use of flawed proxy measures of quality:

• On the EMBO Young Investigator and Installations Grants application forms, in the “Other relevant information” section, candidates can list their involvement in scientific community work: mentorship, outreach, and any other engagement.

6. DORA is advocating for a more inclusive academic community, and supports practices that address structural inequalities in academia, e.g. where there are measures to mitigate bias:

• EMBO is committed to gender balance and equal opportunity. EMBO has implemented many measures, in order not to discriminate or disadvantage specific groups.
• Conscious bias: EMBO has a clear conflict of interest policy, sent to reviewers before each selection meeting. Depending on the level of conflict, reviewers do not take part in an application’s discussion or cannot review the application: https://www.embo.org/about-embo/leadership-and-governance/committees/embo-conflict-of-interest-policy/
• Unconscious bias: EMBO has developed slides to make selection committees aware of the danger of unconscious bias and what can be done to mitigate it. The slides are regularly presented by the programme managers at the beginning of each committee meeting. They will be available soon on the EMBO website.
• EMBO Young Investigators and Installation Grantees: For female scientists, eligibility cut-off date is extended by one year per child and for male scientists by three months per child or by the amount of paternal leave taken (whichever is more). Any other care responsibilities are also considered.
• EMBO Young Investigators and Installation Grantees: Childcare grants for conferences. Recognising that taking care of children adds financial and time obligations to parents, EMBO provides additional support for scientists with families. Eligible costs include fees for a baby-sitter or child-care facility, travel costs for a relative or friend who can take care of the child, or travel costs for taking the child to the meeting etc.
• Young Investigators and Installation Grantees: For female programme members, a one-year extension per child born during their programme membership is granted, i.e. extension of benefits such as networking and support of their lab. For Installation Grantees, this means an extension of programme benefits and longer to spend the grant funding.
• Postdoctoral Fellowships: Dependent child allowance and parental leave are granted.

7. Transparency:

The names of the selection committee members are published on the EMBO website, as well as the selection criteria for each funding scheme: https://www.embo.org/funding/fellowships-grants-and-career-support/

8. Other efforts toward responsible research assessment:

Training in research integrity:

• EMBO grantees must take research integrity training as a condition to receive the funding. EMBO provides an online course on research integrity for free.
• The Young Investigator Programme provides free-of-charge training courses in research integrity for PhD students in EMBO Young Investigators’ labs.

Peer review, both for grantmaking and for scientific publishing, is the cornerstone of quality control in science. Researchers carry out this activity as part of their contributions to scientific research. EMBO is driving discussions on how to recognize peer review in the assessment of researchers.

Most recently, EMBO has engaged in recognizing peer review for preprints, in a meeting co-organized with HHMI and ASAPbio Recognizing Preprint Peer Review , 1-2 December 2022. The meeting was covered in an article in Science.

EMBO organizes and leads sessions at international conferences on issues relating to research assessment and peer review.

A selection of upcoming events:

Exploring best practices in research assessment: EMBO, EMBL and DORA, 12 May 2023 a hybrid panel session to present and discuss funders’, institutes’ and publishers’ initiatives to honour the DORA principles.

A selection of past events:

Recognizing Preprint Peer Review, a meeting co-organized with HHMI and ASAPbio, 1-2 December 2022. The meeting was covered in an article in Science.

Experiments in evaluation: lessons from randomisation in research funding. An SNSF, EMBO & RoRI virtual workshop
1 and 2 December 2021

Understanding peer review and government grant selection process, 6th Annual LymeMIND conference, 23 October, 2021. Video of the session

Setting Positive Incentives in Research Assessment: How to align criteria with research integrity principles, ENRIO 2021 Congress on Research Integrity Practice, 27-29 September 2021

Emerging Trends at the Intersection of Research Publishing, Open Science, and Academic Reward and Recognition Systems, CellBio2020 Virtual Meeting, 2 December 2020

Addressing Roadblocks in Research Assessment Reform, EuroScience Open Forum (ESOF), 3 September 2020

How to Improve Research Assessment during the Triage Phase of Application Review, ASCB | EMBO Meeting, December 2019

EMBO is part of the Open Science task force of the Initiative for Science in Europe, which published the report “Centrality of researchers in reforming research assessment – Routes to improve research by aligning rewards with Open Science practices” in February 2022

EMBO is a partner in RoRI, an international consortium of funders, universities and technologists. The group uses evidence-based approaches to experimentation to improve the way research is funded, practised, and evaluated to ensure that it benefits more people. 

The EMBO Policy Programme is involved in the RoRI workstream on randomisation.

Read the The experimental research funder’s handbook produced by EMBO with RoRI, the Swiss National Science Foundation (SNSF) and Innovation Growth Lab (IGL) at Nesta.