Lawrence Rudner

In contrast to the pariah status accorded to coaching companies in India by organisers of entrance tests, the Graduate Management Admissions council (GMAC) has been meeting coaching companies across the world to introduce to them the new version of the Graduate Management Admission Test (GMAT) with its Integrated Reasoning (IR) section slated to be launched in June 2012.

PaGaLGuY talks to Lawrence Rudner, Chief Psychometrician of GMAC to find out how the organisation makes peace with test-prep companies and how GMAT applicants planning to take the test after June 2012 should prepare for the IR section.

While working with coaching institutes to introduce the IR section, aren’t you worried that test-prep companies might learn tricks that may end up compromising the test?

Traditionally test-prep companies and testing organisations have been enemies. Testing companies see test-prep companies as the ones who teach tricks and test-prep companies see testing companies as sneaky people who develop nasty questions.

But if we step back, we have the same goal of helping students demonstrate what they are capable of. The way I look at it, I would be an authority on the science and psychometrics of the GMAT while coaching schools would be experts on the art of the GMAT. They know the questions better than anybody. So, personally it’s a great group to have a conversation with. It’s also very productive. We are both working for the same goal. They are looked at as the ultimate knowledgeable authority by students and it is in everyone’s interest that we talked to coaching companies and gave them the most accurate and updated information about the GMAT as possible.

But do you think that coaching institutes ultimately weaken the GMAT by affecting the candidate’s ability to perform in the test?

I will answer the second part of the question with a yes. You need to be prepared for any test. The GMAT is unlike most other tests people have taken before. For example in data sufficiency, we are not asking people to solve a problem. We are only asking if there is enough information here to solve the problem. So, you only have to answer a question as opposed to solving the problem. Although some tests in India have modelled some of their questions around data sufficiency, most students don’t have any experience with such questions. So if you went to a test cold, without ever looking at the material or ever studying, then you will instead solve the problem as opposed to answering the question.

Similarly in sentence correction questions, we are not asking which option is grammatically correct. We are asking which one is grammatically correct and provides the best way of expressing the concept.

I believe that everybody needs to prepare and get their pacing correct so that they are spending the right amount of time on each question. Some people need coaching companies to provide that discipline. Others can do it on their own. But my key point is that some people need coaching and some don’t.

So where does the influence of coaching end towards preparing a candidate to game a test like the GMAT?

Because of all the science that we have behind the GMAT and the care we put into writing our questions, I am not concerned about any tricks being taught to students. There are a lot of sloppy ways to write a question. For example, of the four options, one can be really long and the others short and the long one would be the right answer. But we go through so many reviews and take so much care to write our questions that I am not worried about coaching institutes formulating tricks to beat the GMAT. Preparing for the GMAT is really about getting familiar with the questions.

But as you get better at writing the questions, the coaching institutes too would be getting better at identifying loopholes to formulate tricks. Is there a constant race between the both of you?

There are no tricks. Even if they do identify tricks, we find it out and do something about it. But I think that the GMAT is one test where there are no tricks. We are familiar with what test-prep companies offer because we ourselves offer GMAT test-prep products.

A lot of people spend time trying to analyse and psyche out the GMAT. For example, they think that because they got this question easy therefore the last one must be wrong. People should realise that they are spending valuable time ineffectively by analysing the test this way. Students are not a good judge of whether a question is easy or tough. What might be easy for one person might not be easy by the average standards of all GMAT takers, or the question might simply be from an area they are more comfortable with. In fact, even test question writers are not good judges of ease or difficulty either.

How is the run up to the launch of the new format of the GMAT coming along?

We did a survey of b-schools in which we asked how management education was changing and what new skills would b-schools want in future students. They answered that they wanted to see students with higher analytical skills where they are taking information from different sources and combining that to make knowledge. We listened to the schools and started designing the IR section. We started with 12 different structures of questions and eventually narrowed down to four. IR tests a skill relevant for management education in today’s world. A lot of b-schools get criticised for being factories producing clones of the same product. What businesses are looking for is people who can think and analyse a wide variety of perspectives and information.

In the first phase we will have IR as a section of the GMAT. I don’t know what the version after that is going to look like, but we will know after we have collected a lot of experience.

One difference in the way we introduce these changes compared to how other testing agencies do is that we do it slowly. This gives us, students and b-schools the time to get some experience with the new section. First everyone will only have the scores of this new section as an increment to the older verbal-quant-total-AWA score. We will then make an informed decision about further remodeling the test. This method of bringing about change is as opposed to throwing out the older test completely to get the new one and taking a big risk assuming that the new test is going to work.

So, would IR eventually grow to replace other sections of the GMAT in the future?

Yes. One of the things we are looking at is to know how the existing sections correlate to this new section. A lot of our existing questions are tapping somewhat the same skills as the IR. I would not be surprised if someday in the future there would be more and more IR in the GMAT if not in the whole of GMAT. But we will tread slowly there. We have a well accepted test with which we don’t want to be taking risks with.

Would it change the type of students schools are getting?

In the beginning, it will only be a differentiator during admissions. A school might say that here are two students who have the same quant-verbal-total-AWA scores but one student has an additional metric. Later, this has the potential for b-schools to get students more like what they are looking for as opposed to just unitary analytical skills.

During the transition phase from the current GMAT to the new GMAT, b-schools would get applicant pools with both types of scores. How do you think they would reconcile between the two pools?

Well, it’s fairly easy. Older students will have quant-verbal-total-AWA scores, newer will have quant-verbal-total-AWA and IR scores. If anything, IR will give some students an advantage, they will have an additional data point they can use. But since GMAT scores are valid for 5 years, this overlap will be there for the next 5 years. But the overlap will reduce over time. I believe that students who have the IR score will have a better opportunity to demonstrate what they are capable of. I would personally go and take that test again in June.

Since your research shows that schools are indeed looking for IR ability, would it be right to say that those who apply with the older format GMAT scores will actually be at a disadvantage when applying for the 2013 intakes?

That’s way too strong. Those that do have IR have an additional data point. How schools behave is something I can’t predict. History has shown that when you release a new score, schools are very slow to use it. I wouldnt be surprised if they first want to look at it, get a certain experience with it, look at how the high scorers in the IR section are performing in the classroom before putting more and more weight on it.

So initially, I don’t think it to be much of a disadvantage, if at all. But a year or two down the road, it might be a disadvantage because if there is a tie-breaker, the school will go with the applicant who has the IR score. But by then most students would take the GMAT with IR. But for now, disadvantage is way too strong a word.

How should candidates prepare for the IR section?

Get hold of the questions online as well as on the prep material we would be releasing and practise. Your goal should be to be very comfortable with the questions and understand what we are looking for. In essence, there isn’t something they should do very differently compared to what they do for other sections.

As the launch date of the new GMAT comes closer, is there a race among candidates to take the test in the older format that they are comfortable with?

Normally, 80% of test takers sign up three months prior to the test date. So if at all there is a race, we will not know before February 2012. But as of now the registration for the June test is on and we don’t see any racing yet.

What kind of numbers are you seeing sign up for the June 2012 test now?

I don’t think the new section is going to be a big draw for students at least initially. So, I suspect that what we are getting now is the usual set of people who sign up 6 months in advance.

Why is the IR section not adaptive?

For being adaptive, you need a really big pool of questions. We don’t have that yet. We are building it and eventually it would be adaptive but not now.

What is your recommendation for an applicant who wants to take the GMAT for the 2013 intakes — should he take the older GMAT or the newer one?

I recommend you to just take the test when you are ready and comfortable.

What interesting trends do you see among Indian GMAT takers and b-schools?

Schools in India now get the fifth-largest number of GMAT test scores in the world. Indian b-schools are now in the second position as far as getting scores reports from Indian applicants are considered, even more than UK b-schools. So, India has surpassed UK as the destination for Indian GMAT takers. US b-schools still get the largest number of Indian score reports, but it has been on the decrease.

The percentage of women GMAT takers in India is embarrassing at 21%. In China, it’s the other way round; 60-70% of the test takers in China are women. We can conjecture why that happens. In China with its one child policy, women are the same as sons. That is not the case in India. We also see a wider range of people in China take the test. In India it’s mostly the engineers who take the GMAT whereas in China there are as many people with humanities who take the test.

One thing I don’t understand is why we don’t see as many fresh undergraduates in India taking the GMAT as we do in the rest of the world with an aim to get into Masters in Management and Masters in Finance type of courses.

Are you planning any differential pricing of the GMAT for Indians?

No. Our issue is that we are investing a lot of money in the GMAT. Not just the test and its quality, but also in the services we provide, from the various researches we do, even the custom research for b-schools. We are not just about volume and revenues, but also valuable services for the management education community.

Can you comment on how computer-based testing in India is evolving?

From what I understand, they are making the tests here less of a timed test, which is good. The issue I have with timed tests is that while they do a good job of identifying the top 2%, they are not that accurate at identifying the abilities of the next 5% or 6%.

Indians don’t seem convinced that our computer-based tests are evaluated fairly.

I can’t comment much about their methodology. I have my suspicions about how they are doing it, and I doubt their comparability from one form (question paper) to the another. I do have a problem with the test being a new race every year. As you know, in GMAT the scores are good for five years and the scores always mean the same thing. We do publish percentiles which we update every year. So if the competition was stiffer this year, your percentile for the same exam would have gone down.

What is the future of testing?

The next big breakthrough will be a way of doing testing outside test centers. We call it pyjama testing. Somebody will figure out a way to do secure high-stakes testing in a non-secure environment and that will be a big breakthrough. There are some people playing with it and we are watching what they are doing.

I’m conjecturing that in 10 years, the science of doing diagnostic testing at the same time as doing high-stakes testing will get better. We as of today have a diagnostic test but its fairly crude. It gives you very broad characteristics about yourself but in the future we will be able to give you very pinpointed diagnostic information. Apart from this, there will be testing in much more interactive scenarios, like role-playing in games. Testing will also be embedded in the curriculum.

Also read: What differentiates a good test from a bad one

Read More