What is the attainment gap? With Ricky Ellis

[00:00:00] Mat: Welcome to Doctors at Work. My name is Mat Daniel, and this podcast is about doctors’ careers. Today’s topic is differential attainment. I’m having a discussion with Ricky Ellis, and he will share his work with me. It is a fact that doctors from different backgrounds have different success rates in postgraduate exams and in interviews and different rates of representation in senior leadership.

In part one, Ricky tells me about his work on differential attainment, and we discuss what it is and how it arises. The next episode of my podcast will be the second part of our discussions, where we will discuss what we can do about it. I hope it’s useful.

Welcome, Ricky. Tell me a little bit about yourself.

[00:00:53] Ricky: Thank you. Firstly, thanks for having me. I feel quite honoured to be invited. I am originally from London. I started my educational journey by studying biomedical sciences. After which time I went to Canada for a while and worked in research and development and then came back to study medicine at Warwick.

After which I went through the academic foundation training program up in the East Midlands and I’ve stayed here since Falling in love with these Midlands and we’ve got the Peak District on our doorstep. So, I can’t see me moving out anytime soon And I’ve got a real interest in medical education recently completed a PhD in medical education While I was doing my urology training in the East Midlands

[00:01:35] Mat: And I’ve invited you to talk about differential attainment or the attainment gap.

How did you get interested in that area?

[00:01:43] Ricky: Yeah, it’s a great question. So, I was lucky enough to be awarded a research fellowship by the Royal Colleges of Surgery in the UK and Ireland. And my, the description of my research was going to be investigating the validity of the membership of the Royal College of Surgery examinations.

And actually, it was while I was looking through the data and trying to analyse and apply the data that I was working with, I started to notice some huge discrepancies in the data. Really, I started to notice huge differences in past marks between different groups of surgical trainees taking the exam, and it was harrowing differences in past marks that, made me investigate it further.

I started reading around the subject and I looked at the work of my predecessor Duncan Scrimgeour and we just started to identify this huge difference in attainment between different groups of individuals sitting the exams and actually you know as you start to read about this that it’s a problem that exists not just in surgical exams and not just in postgraduate examinations either it’s present in almost all assessments that we do for our entire educational careers and that has huge consequences for career progression for some groups of individuals and not others.

[00:03:05] Mat: Okay. So, it strikes me that this won’t be news to some groups of individuals, but it’ll be quite surprising to other groups of individuals.

[00:03:14] Ricky: Exactly that. Exactly that. For many, this won’t be breaking news. And actually, we’ve known about this for decades. There was a fantastic paper by Professor Esmail about 30 years ago now that really highlighted that the degree of bias that exists within recruitment processes within assessment.

And, you have to ask yourself how much has changed in the last 30 years. The answer is. Not that much if, if we’re still seeing this data, these, this, systemic differences in performance between different groups of individuals on the same assessments, we still have a huge problem that needs addressing that has been neglected. I think over the last few decades.

[00:03:56] Mat: And I’ve got to ask how did the Royal Colleges react to your findings.

[00:04:02] Ricky: Yeah, it’s, I would say it’s. A tough read. Anytime you see this data, it’s very difficult to read. You are reading about your colleagues, your friends, your peers, having to work twice as hard to perform as well as you on the same assessment.

And that’s not because, they, their skills on their knowledge is less than yours is because of other factors outside of their control. So, I think it’s a really difficult thing to read for anyone. It’s difficult. topic to discuss. It’s a difficult topic to approach. And doubtless, it’s a difficult topic for, the exam conveners and, people responsible for setting assessments in, in medical training programs.

It’s a big conundrum. It’s a big problem. I can’t, I can imagine that it was a difficult read at all levels including for, council members and college presidents at the Royal Colleges of Surgery.

[00:04:59] Mat: So, what is the attainment gap then?

[00:05:03] Ricky: What we talk about when we mentioned attainment gap, or the awards gap or differential attainment is systemic differences in performance between two or more groups of individuals.

At the same assessment. So that can be groups differentiated on protected characteristics or socio demographic or socio-economic factors. And it’s really important. In terms of understanding this to know that these aren’t differences in performance on an individual level, this isn’t due to some learners having not studied as much or their skills and knowledge not being as good.

These are differences in performance between huge cohorts of individuals, some of my papers have used cohorts of 11, 000 surgical trainees. These are huge numbers so it’s not due to learner deficit. This is due to other systemic factors causing this difference in attainment.

[00:05:59] Mat: Okay, and can you tell me more about the systemic factors?

[00:06:04] Ricky: Sure, in terms of what causes it it’s a very, the causative factors are likely to be very complex and multifaceted. It would be very easy to say that the problems only exist within the assessment themselves. That would be the easiest thing because it’s very easy to correct that, or it’s easier to correct that. But sadly, I’m not going to hang everything on that hook because, firstly bias and discrimination has been ruled out of several postgraduate exams used in the UK where differential attainment has been identified.

Secondly, it also exists in written components of exams and assessments. Bearing in mind that these are marked anonymously by computers, so that can’t be put down to examine a bias or discrimination, for example. In terms of the other reasons that might be behind it to use a bit of a concept taken from elsewhere in education, Assessments are the lens through which we see the accumulation of social and educational opportunities and historical bias and discrimination that people experience throughout their lifetimes.

So, what we’re seeing actually is, the accumulation of privilege and disadvantage that results in some people having headwinds, holding them. Back, in assessments and in career progression, while their colleagues are experiencing tailwinds that propel them in their journey to success.

If we think about it from that sense, it’s actually very hard to identify the causative factors. We can break it down into problems with the assessment, problems with the assessment methods, the possibility of bias and discrimination within the examination. But actually, I think that the thing that’s most interesting is the differences that might exist before the assessment within the learning environment. Because actually they’re the factors that accumulate and are expressed in differences in attainment at the assessment itself.

[00:08:13] Mat: I hope you’re enjoying the show. If you are, please click subscribe so you will be notified when new episodes come out. This podcast is part of my mission to help doctors create successful and meaningful careers.

You can be part of that mission too by forwarding this show to any one person who you think might benefit from listening. Thank you. Now on with the show.

Okay, so this strikes me as very interesting and also very, philosophical, because I was expecting you to tell me that people just discriminate, that, you go to an exam and it’s full of middle aged, white, straight men, and they like people that are, other white men, middle class from public schools, and they discriminate against anybody who isn’t.

That’s what I was expecting you to say. But certainly, when it comes to computers, unless the computer went to a public school and is white and middle class and male, which, it might have done, or it might have been written by white middle-class men. But unless it did, then what you’re telling me that actually, yes, that might be one part of the problem, but there’s much, much more to it than just that discrimination that happens at the point of sitting the exam itself.

[00:09:21] Ricky: Absolutely. And we are more aware of this now. I say how much has changed in the last 30 years. Selection into medical school is a really good example of that. There was a huge amount of differential attainment in selection into medical school. If you look back at a picture of a lecture theatre, a medical school lecture theatre from, 50 60 years ago, it’s generally all white men.

So, we’ve known about this for a long time. And thankfully, we have moved away from more nepotistic recruitment methods into more, nationalized selection processes that generally involve interviews with multiple people. And. And you can see that does have positive results. We do have a much more diverse workforce now than we ever had before.

For example, we now know that last year, more than half of all UK graduates were female and more than two thirds of the UK workforce are from black and minority ethnic backgrounds and more than 10 percent have a registered disability. These numbers are a huge improvement on what it was, if you look 10, 20, 30 years ago, for example, that isn’t enough.

You, we can’t be satisfied with diversifying the workforce. If some groups experience inequity in training opportunities and in assessment. Therefore, preventing their holding back their progression compared to their colleagues throughout the rest of their careers, and this is why ultimately, we were talking about lack of diversity in senior and leadership positions within medicine, and it’s very topical at the moment. How do we expect to diversify the senior and leadership positions within medicine when barriers such as these assessments exist? For some groups and not others preventing their progression,

[00:11:15] Mat: It, it strikes me as perhaps maybe it’s a bit odd that here we are two white men talking about this. Why should white men care about this?

[00:11:24] Ricky: Yeah, so it’s a really good question. And one that I get asked a lot, actually I think it’s really important that, there’s an, there’s a lot of privilege that comes with being a white man. It’s undoubtable, and especially within medicine as well, especially within surgery, and it’s important that we recognize that privilege, that we have insight into that privilege, and also that we don’t sit back and expect, all of the burden of this work of trying to improve the culture of training to try and improve, progression for everyone, equity within training opportunities.

We don’t let that burden fall solely on the shoulders of people that have already been minoritized by the system. There are people that are having to work twice, three, four times as hard as others to be equally as successful within medicine. Now, if they have to shoulder the burden of changing things for future generations as well, that’s an awful lot that’s on their shoulders.

I think ultimately, this is going to require a top down and bottom-up change by everyone and contributions by everyone. So maybe you’re lucky enough for differential attainment never to have affected you. And in which case, that’s incredible. And also incredibly privileged position to be in because, this has changed the lives of many other people around you, and this affects their lives on a daily basis.

So, if you are lucky enough to be one of those people, one of those privileged few that hasn’t been affected by this, then absolutely feel that it’s a duty to, to get involved and to try and help where possible. And it’s our duty to, to listen to how we can help if, if there’s something you can do, why wouldn’t you try and help your colleagues your peers, your friends, family, because it really is unjust that people are having to work twice as hard to achieve the same thing.

[00:13:26] Mat: The thing that really struck me as important and I clocked this having a discussion with. One of my 60 plus years old colleague, a number of years ago, and I can’t remember how we got on the subject and he’s a white male professor. And he doesn’t discriminate against anybody, I’ve known him for 20 plus years. He’s very fair, very equal to everybody. At least from my perspective! And we were talking about discrimination and what became obvious to me is that he didn’t think that it existed because it’s not something that he did. Yeah, so he didn’t do it, and I don’t think that he did it, and nobody’s accused him of doing it, but because he didn’t do it, he didn’t see that it exists. He didn’t do it, and it doesn’t affect him, therefore it didn’t exist. Yeah, and that’s problematic, and maybe that’s one of the things that that I’ve clocked for myself, is that if I go into the workplace and say that we talk about, gender discrimination, and of course I don’t see it because I’m male.

And maybe for other people in the workplace, I’m gay and other people in the workplace, they might not see LGBTQ discrimination, but I do. And of course, other people don’t see it because they’re not gay. And, and if you’re white, of course you don’t see an ethnic discrimination because you are in the white majority. So, it’s that a bit of a bit of being open to, to recognize that, okay, this doesn’t affect me. I don’t see it, but just because it doesn’t affect me, it doesn’t exist. So, the starting point is that unfortunately this exists, whether you see it or not, or whether you believe whether you experience it or not, it does exist.

That, for me, then, is a really good starting point that we stop pretending that it doesn’t exist, and we say, yes, it does exist. It doesn’t affect me, maybe, but it does exist, so let’s do something about it.

[00:15:11] Ricky: Yeah, absolutely. I think the key here really is that having the ability to listen to others to because that will help you recognize that these issues do exist, even if you’ve been privileged enough for them not to affect you personally and recognizing that everyone’s lived experience is very different.

So, we don’t, we talk artificially about differences in attainment, for example, in between groups. And we talk about differences in experience between groups of individuals, and often they’re broken down by social demographic factors or protected characteristics. It’s also important to remember that we don’t just belong to one group.

So, everyone experiences. opportunities and disadvantages, privileges, biases, and discrimination differently. There’s talk of, double, triple, quadruple jeopardy for some groups, if you are a black female that works within surgery, you’ve, it’s double jeopardy looking at all the data, if you are from a black and minority ethnic background, you’re less likely to pass your exams and you’re, if you’re a woman, you’re less likely to pass your exams, you’re less likely to progress for your ARCPs, your appraisals, so you’ve got double jeopardy there. So, everyone experiences these disadvantages and privileges, on a different level and it’s being able to be open to listening to that and recognizing your privilege and, the experiences of others. I think that’s really key actually.

[00:16:39] Mat: Okay, so what does the data show then?

[00:16:43] Ricky: The data show that the data. harrowing actually, it’s an awful read to be honest with you. The, my work revolved mostly around the MRCS, the membership of the Royal College of Surgery examination. And when I looked at the 10 years’ worth of surgical trainees taking the exams, we found that if you were male and if you were a younger trainee, so if you’d studied medicine as an undergraduate, for example you were twice as likely to pass the written component of the exam. And that’s after adjusting for all other demographic social demographic factors and adjusting for measures of prior attainment, a proxy for academic ability, really. So just based on your gender and your age, your likelihood of passing was significantly different. And, if we think about the clinically remember as well, that’s anonymously marked, like I say, a written component of the examination with the clinical examination.

If you were younger, and if you were from a more privileged socioeconomic or educational background, so for example, if you went to a fee-paying secondary school, you were twice as likely to pass the clinical components of the exam at the first attempt. And if you’re from a black minority ethnic background, you are half as likely to pass the clinical aspect of the exam at your first attempt.

And this isn’t just exclusive to the MRCS. This exists in almost all postgraduate examinations taken throughout the UK, the US, wherever it’s been studied, differential attainment has almost always been identified. For example, we know that men have higher pass rates in all surgical exams. They have higher pass rates in some anaesthetic exams.

Women have high pass rates in general practitioner membership examinations membership examinations for psychiatry and MRCP clinical examinations. So, you can see these differences across every exam. And it’s not just on gender either. We can see differences in age, socioeconomic factors, but the one that really stands out is differences in attainment according to ethnicity.

White candidates. have significantly higher pass rates in every postgraduate examination taken in the UK, and most that have been studied across the world. And in fact, Professor Wolf and her team in 2011 did a fantastic paper, but it showed that if you are white, you are three times more likely to pass an assessment in medicine than your colleagues, just based on your ethnicity.

When you see data like that, it just shows you how big this problem is and how much this is affecting people’s lives. And it’s not just for exams. It exists in all other assessments, in appraisals, in ARCPs. As well, there’s really good evidence in surgical ARCPs that the women, the older trainees, those from less privileged backgrounds are less likely to pass their appraisals and progress in their careers.

If you’re from a black and minority ethnic background, you’re 40 percent less likely to pass your appraisal in anaesthetics. This differential attainment exists across your entire educational career. If you think about moving for exams and moving for a yearly appraisal, moving through selection for higher specialty training, if you’re suffering.

If you’re experiencing differential attainment in each of those, how many barriers exist to progression in medicine or medical careers for some groups and not others?

[00:20:47] Mat: If I go back to the causes then, because you’ve told me that just discrimination at the point of assessment, that might be one component of it.

But there is more to it than that. So you said that it’s complex, but how does it happen? Because you’ve corrected for academic achievement. So, say we got, presumably you did what the grades at A level or grades at medical school. You take, you correct for grades at medical school and grades at a level and you still left with a difference based on ethnicity or gender or age. And you, I know you said that it’s complex, but how does that arise?

[00:21:24] Ricky: Sure, so if we think about let’s think about what we can control. So, let’s think about medical training. So that’s when people are Under our wing as such if we think about what could affect performance in assessments within that time, I like to break it down into hierarchical system.

So, you’ve got on a systems or policy level, there can be bias within policies, there can be structural discrimination. And we talk about the hidden curricula as well which affects individuals. Then on a more regional, institutional level, you can start to think about things like access to role models, access to coaching, to mentorship, these are things that help you navigate training. They help you navigate careers and the career pathways. They help you navigate assessments and learning as well, and differences in, in opportunities in each of these will have a significant role in attainment in performance.

If we think about on a still on that level in flexibility within training programs as well, we know that the training certainly within the UK is very constrained. And it’s, let’s be honest, it’s created for people who, aren’t caring for dependents, don’t have to rush home for children, aren’t disabled, aren’t less than full time. It’s, it works relatively well for people that don’t have any of those responsibilities or those differences. But as soon as you start adding, caring responsibilities, children, disability, things like this, it becomes very inflexible. Also, everyone learns differently. The training program assumes that everyone learns the same. Everyone can only have study leave to go on a course. They can’t have study leave to just revise at home. Everyone has to take these courses to learn. They can’t do it in their own way. It’s very inflexible. And then if we think about on a micro level, so a personal interpersonal level, you’ve got things like bias, microaggressions, bullying, discrimination, and harassment that people are experiencing every single day.

And it’s inevitable that will affect experiences within the learning environment. All of that adds up to a term called social capital. If you have less social capital within the medical training environment, you may feel a lack of belonging. Your networks are smaller. You then, it’s a knock-on effect. You have less access to role models, whether they’re your level or maybe a high level in the training program or career. You have less access to colleagues, to revision groups, to help you navigate the assessment, the learning pathways. So, if you think all of these factors on a personal or interpersonal level are affecting your social capital, and that has a direct impact on your performance.

These are just a few examples of differences within that training environment, within the learning environment that, that, we have under our control. We can change all of those factors, there’s myriad other factors that we may not be able to change because it either happened before they entered medicine, it may happen out of our control so at home, or in their social lives, but these are things that we can absolutely change and absolutely address and clearly have a significant impact on performance.

[00:25:14] Mat: Yeah, I love the idea of social capital because it, it captures the assets that somebody has and has access to versus another person that doesn’t have those or doesn’t have access to them.

I hope you enjoyed the episode. I don’t know about you, but there’s some quite shocking statistics in there. In the next episode of my podcast, we’ll move on to having a discussion about what we can actually do about differential attainment. So have a listen to that episode as well, and hopefully Ricky will tell us some tips about what we can do to make things better.