Our Research Manager, Desh Mofidi, offers his insights to this question based on his experience working for five years at PRUK, following three years working in the National Institute for Health Research’s (NIHR) Research for Patient Benefit (RfPB) funding programme.
While I was working for the NIHR RfPB programme as its Programme Manager before joining PRUK in May 2016, an interesting exercise was carried out whereby one application was sent to all the regional panels for review. This exercise was carried out blindly, so panel members were not informed until after all reviews had taken place. The purpose was to see whether there would be consistent reviews across all panels and if not, determine the reasons why this would be the case. This exercise was an eye opener for me, as I learned just how important guidance and planning was when assessing applications. It also points to a question that I have received several times: “Are panel’s biased when they review applications? Is the process truly transparent and fair?”. This is an important question and one that, as Research Manager, I feel better equipped to address.
Firstly, there are many factors that are considered when applications are being assessed. Principally, the scientific rigour and robustness of the application is scrutinised. This will involve reviewing the research question, its relevance and whether the applicant has considered the most relevant and latest research from the field. They should have justified how their proposal would add to the current literature. The aims & objectives of the proposal would be assessed to ensure they are relevant to the research hypothesis and that the outcomes are relevant with the aims of the research. Another key factor (where the panel tend to criticise the most) is the methods section. Often applicants fail to appropriately justify the use of a particular methodology and do not consult experts to ensure the methods have been clearly outlined in the proposal. Patient and public involvement (PPI) is another critical area, where funding bodies have dedicated PPI representatives to assess applications and it is therefore paramount that applicants consult with patients in the development of the research plan, application process and throughout the course of the research itself (not just the plain English summary). For PRUK, the future career aspirations of the applicant are crucial, as we are dedicated to ensuring that we can create the pathway for pharmacists to develop their research skills and knowledge to allow them to become established researchers.
While the panel are always focussed on ensuring the application is reviewed robustly and only the best applications receive the funding, all applications are assessed in competition with other submissions in any one round. This means that applications are essentially competing to receive the pot of money budgeted, so it becomes even more important that the best applications in that round are offered the funding. Panel members need to also consider the budget available, as this can play a role in how many applications are funded. There are therefore multiple factors in play when assessing applications, but at the heart of the review process are questions relating to the points made in the previous paragraph. Panels will constitute of members that have expertise in research across a range of different disciplines. Whilst it is not possible to have a member that is an expert in all relevant areas, members are recruited based on where the most need is for representation. This is why external peer review is such an important process, as it helps to bridge any gaps that may lie in subject knowledge.
The panel are then asked, after doing a full assessment of the application, to score the application and any proposal scoring above a certain threshold (i.e., above not fundable) will be invited for further discussion at the main meeting. This is where I have developed further guidance where the assessing members are expected to present their review, considering all aspects of the application, the applicant, and the external peer reviewer comments. We also ask PPI representatives to present their reviews and it then opens to further discussion from the panel before a final round of scoring takes place. The guidance that we offer to our members ensures that their reviews are as consistent as possible, as it must follow a certain order and all aspects of the application are covered systematically. Members are provided with templates so that we can best ensure this happens. Conflicts of interest are also gathered from submission of the application, through to the review of them and these conflicts are declared to the panel and PRUK.
Despite all the processes we have implemented to ensure applications are reviewed with rigour, transparency, and fairness, of course there will always be variation and disagreement because applications are assessed by human beings, and we keep our processes under ongoing review to make sure we can identify and respond to any issues. We are confident though that we have a very robust approach, that gives us the best possible chance to make fair and accurate decisions. And this is very much thanks to our panel members, our reviewing procedures, and the academic researchers who peer review for us.