Linking data to get results: India shows how countries can use their national assessments for global reporting

Por Silvia Montoya, Director of the UNESCO Institute for Statistics (UIS) and Professor Hrushikesh Senapaty, Director, National Council of Educational Research and Training of India (NCERT)

The international reporting of learning outcomes is a critical but complex undertaking at the global, regional and country levels. Yet in just a few years, we have made remarkable progress in resolving measurement challenges associated with SDG Indicator 4.1.1: “proportion of children and young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving a minimum proficiency level in (i) reading and (ii) mathematics, by sex.”

This is why the Inter-agency and Expert Group on SDG Indicators (IAEG-SDGs) recently decided to upgrade 4.1.1. to a tier 1 indicator: the Group recognized that the indicator is “conceptually clear, has an internationally established methodology and standards are available, and data are regularly produced by countries for at least 50 per cent of countries and of the population in every region where the indicator is relevant”.

This upgrade was no easy feat. Just a year ago, we went through the first phase of the process, by presenting a new methodology and standards. This involved painstaking negotiations with countries and just about every cross-national assessment initiative in order to build consensus around a set of global minimum proficiency levels.

Increasing the use of national assessments to monitor quality education

To monitor learning globally, we must overcome three main challenges to compare data from countries with different assessments. First, the assessments do not always use the same definitions of proficiency. Second, we need consistent quality, based on shared technical standards, to ensure that national and regional data are fit for purpose. Third, we must accommodate multiple viewpoints, which means: identifying relevant areas of learning that can and should be measured globally; conceptualizing how national and regional data can inform global measurement; and striking a balance between global perspectives on education and local influences and goals.

The best way to address these challenges is by linking up national assessments in order to express them on a common scale. There are two main ways this can happen: statistical and non-statistical. Statistical linking is generally more accurate but also more complicated and expensive to apply (e.g. either the same students take two or more assessments or common items are embedded in two or more assessments in different countries through a new data collection).

In contrast, non-statistical linking makes it possible to use existing national data and avoid the need for a new data collection. While this approach is generally less accurate, experts argue that it is acceptable when statistical linking is not feasible because of methodological, operational or financial constraints. In particular, policy linking (which is one of the non-statistical approaches) allows us to link one assessment to another using the same set of descriptions of performance levels.

We are convinced that the new policy linking methodology will engage countries and help build statistical capacity. We also know that every new tool can be further refined. By testing the methodology, we may learn more about how pre-existing conditions can impact implementation, what benchmark methods to use, and how sensible the results are in order to reduce the risk of any possible bias in international comparisons and reporting.

This is why, the UIS, with the financial support of the Bill & Melinda Gates Foundation and the UK Department for International Development (DFID), is organizing a series of pilot tests in countries, such as Bangladesh and India, where millions of children are making their way through the countries’ vast education systems.

Working with teachers and curriculum experts in India

India is committed to education reform based on hard evidence and is eager to contribute to the work of the Global Alliance to Monitor Learning (GAML). The UIS and India’s National Council of Educational Research and Training (NCERT) are working today, in the first of two workshops, to apply the new policy linking methodology to national assessments for Grades 3 and 5, which cover 1.4 million children from diverse language groups and educational contexts. The workshops involve about 10 curriculum and assessment experts, mostly from NCERT, and 36 language and math teachers.

There are two major steps in the workshops. First, the teachers will map each item used in Grades 3 and 5 assessments to the global proficiency levels expressed in the Global Content Framework (GCF). Developed by the UIS, the GCF is based on the assessment frameworks of more than 100 countries and provides consistent performance standards for pupils in Grades 2-6 in reading and mathematics.

In the second step, the workshop panel (including teachers, curriculum and assessment experts) will assess the knowledge, skills and abilities required to solve each item in the national assessments. Based on this evaluation, they will then match each item to the global proficiency levels to determine the cut off scores that they will use for international reporting.

Finally, the group will review each assessment item again and determine whether learners in each proficiency category would get the item correct or incorrect. The facilitators will then compile and average the scores identified by the panellists and make recommendations to NCERT and the Ministry of Human Resource Development, who will then make a final decision on the benchmarks and cut off scores at the national level that are consistent with the global minimum proficiency levels.

At the end of this process, India will be able to assess how well and how deeply its national standards capture the knowledge, skills and understanding contained in the global definition of minimum proficiency levels. The country will be able to equate these levels to the local context and define the most relevant definition of proficiency at the country level. In short, the country will gain valuable insight for policymaking while benefiting from consistent reporting towards SDG 4.

The UIS is developing a Policy Linking Toolkit with support from the United States Agency for International Development (USAID), which will be available soon and will provide a step-by-step guide for countries and their assessment organizations to use the methodology.

Requirements for effective policy linking

Policy linking is a promising low-cost, relatively rapid, and sufficiently rigorous non-statistical method to compare and aggregate results from different assessments. It can provide lessons learned across countries to guide future programming and improve learning outcomes. In addition, national authorities can use its results for global reporting on students who meet minimum proficiency standards by grade and subject.

Policy linking does, however, require government buy-in and commitment. By hosting these cutting edge workshops, India is confirming its own commitment and setting an example for others to follow.

Fuente: https://gemreportunesco.wordpress.com/2019/11/18/linking-data-to-get-results-india-shows-how-countries-can-use-their-national-assessments-for-global-reporting/