Looking back at the last three years, one new ‘big idea’ has dominated Indias business school admissions every year.
If 2009 was all about replacing paper-pencil assessments with computer-based entrance tests, then the flavour of 2010 was limiting the number of engineers in MBA classrooms and increasing diversity.
2011 is increasingly shaping to be the year when Indias MBA entrance tests begin consolidating towards a Single Entrance Test. Based on the decisions of FMS Delhi (Faculty of Management Studies, Delhi University) and the IIT b-schools before that, the Common Admissions Test (CAT) is looking like it could end up being that single entrance test.
A single entrance test is a sound idea in principle and ought to be in place — it is less stressful for MBA applicants and liberates the faculty of many b-schools from wasting their time on test invigilation. Rising inflation is also making it expensive for b-schools to organize national level testing efforts, the costs of which are being passed on to MBA applicants.
Prominent MBA admission committees in the western world have been using a single qualifying test — the GMAT — since more than 50 years.
The GRE too is emerging as an alternative admission test for MBA admissions worldwide, but all schools that are accepting the GRE also accept the GMAT and there isnt a scenario where an applicant is forced to take the GRE to apply to a particular b-school. GMAT remains the de-facto MBA admissions test, accepted universally.
Both the GMAT and GRE allow applicants to appear for the tests any time of the year and any number of times, so that qualifying for b-school admissions isnt turned into a one-day-performance lottery game.
Both tests have been put through the trial of fire of countless research papers to measure their statistical/psychometric reliability and validity (more so for the GMAT in the business education context), including those by the owners of the tests themselves. The world today knows the extent to which the tests are effective in measuring graduate study aptitude and with what margin of error. These quantitative measures are regularly updated and made openly available to the public in the spirit of any statistical exercise that calls itself scientific and credible. This is how GMAT does it. Here are some independent studies.
(The statistical reliability of a standardized test is the consistency of its result when administered to the same candidate more than once. If the result varies a lot, then the test clearly doesn’t measure that candidate’s aptitude. The validity on the other hand is the extent to which the test assesses the suitability of the candidate for the education he seeks. Validity is commonly measured as the correlation between an MBA class’s GMAT scores and the first year GPAs.)
In comparison, Indias MBA entrance test cousins lag behind by a measure of decades. Except the Management Aptitude Test (MAT), no widely-accepted test offers candidates the ability to appear more than once in a year. (In fact, the MAT also does not allow multiple attempts in the same way as the GMAT. It is merely held four times a year and each instance is an isolated test not evaluated in relation to another.) The IIMs have been making noises about making the CAT a perennial test but that is still a few years away.
The widely accepted CAT, which calls itself a standardized test and is a prominent revenue generator for the IIMs, has surprisingly never undergone a reliability and validity study.
In June this year, we at PaGaLGuY filed simultaneous Right to Information (RTI) applications with the Indian Institutes of Management (IIMs) at Ahmedabad, Bangalore, Calcutta and Lucknow to seek the reliability and validity data of the CAT in the last ten years. While IIM Lucknow and IIM Ahmedabad replied saying that no such data was available, IIM Bangalores answer betrayed a complete lack of understanding about the questions asked.
IIM Calcutta, which is responsible for conducting the CAT this year, sent the following answer,
IIM Calcutta is responsible for CAT 2011 for which the exams are yet to be held. IIM Calcutta does not have any statistical/psychometric reliability and/or validity for rest of the years sought by you.
Each of the aforementioned IIMs have been responsible for conducting the CAT at least once since 2000. According to insiders, a few IIM professors who over time have sought the CAT data to conduct independent reliability studies have always been denied.
Prometric, which has been powering the CATs psychometrics since 2009 has never disclosed reliability or standard error measures of the CAT. On being asked, the agency has been taking the position that such data will be beyond the comprehension of the applicant community and will further increase aspirant mistrust (the reactions to the CAT 2009 technical glitch fiasco being in the backdrop) and therefore will not be disclosed. The “Standards for Educational and Psychological Testing” and the “ETS Standards for Quality and Fairness” which the CAT scoring process states as being supported by also contain clauses relating to disclosures of reliability and standard disclosures. For example, Standard 8.3 of ETS Standards for Quality and Fairness reads,
“Provide information that will allow users of assessment results to judge whether reported assessment results (including subscores) are sufficiently reliable to support their intended interpretation(s). When appropriate, include estimates of errors of measurement for the reported assess- ment results. If users are to make decisions based on the differences among scores, subscores, or other assessment results, provide information on the reliabilities and standard errors of those differences.”
Are CAT-affiliated institutes provided this data to help them make the best possible judgments based on CAT scores?
Standard 8.4 of the same document reads,
“Document the reliability analyses. Provide sufficient information to allow knowledgeable people to evaluate the results and replicate the analyses.”
MBA applicants may not qualify under ‘knowledgeable people’ but does the analysis even exist for the real ‘knowledgeable people’ to easily access and use? Going by the RTI responses above, the IIMs, owners of the CAT, don’t possess it. Is this the level of rigor behind a test that decides the fate of more than 2 lakh youngsters?
Similarly, there is no publicly available information offering an insight into the testing and evaluation standards for any of the other Indian MBA entrance tests — whether FMS, XAT, SNAP, IIFT, JMET, MAT or NMAT. The system at our b-schools has broadly worked on the belief that if a test has difficult enough questions and its products are getting good enough placements (100%, high salaries, etc) after two years, it must be scientifically accurate and acceptable. Now thats a rather loose framework to work with for teachers of an educational discipline that relies heavily on quantitative methods. Here is another case of business schools failing to behave as they preach.
Part of the reason is that Indias psychometrics scientists and professionals have never bothered themselves with large-scale and high-stakes assessments. The underlying testing science for prominent computer-based tests such as the CAT or the NMAT is designed not in India, but by foreigners sitting across the world in the US in Prometric and Pearson Vue offices. It looks unlikely that Indians will take ownership of the testing sciences unleashed on its students anytime soon.
The question being, are any of Indias MBA entrance tests really ready to carry the weight of selecting Indias future managers on their shoulders alone as the single entrance test? Have they benchmarked themselves against accepted world standards of testing and evaluation? If yes, then where are the standard disclosures?
A single entrance test to most major MBA programs that is held only once a year turns MBA admissions into an even bigger lottery game than ever before — perform on that one day or waste an entire year. Wasnt computer-based testing supposed to rescue MBA aspirants from that conundrum in the first place?
Until now, the comfort of multiple entrance tests by various b-schools was the MBA applicants insurance against this lottery-like admissions system. It was stressful for aspirants, but that stress was worth it if one was getting multiple opportunities to demonstrate their aptitude. Now that insurance is also vanishing.
The IIT b-schools and FMS should perhaps have continued with JMET and the FMS entrance test and waited for the CAT to first become an year-round test that candidates can take multiple times before shedding their own tests and embracing the CAT. B-schools looking to shut down their tests and adopt the CAT should do so in consultation with the IIMs and align their actions with the formation of the proposed ‘CAT Corporation’ and subsequent year-round availability of the test.
Instead, they are acting too fast to solve their own problem. Both Prometric and the IIMs have been peddling their inclusion into the CAT fold as a good thing to candidates. The candidates are left to find solace in the ability to choose the day and time they would like to play the lottery.