The Center for Institutional Analysis of Science and Education at the European University at St. Petersburg analyzed the work of the experts from the Supervisory Authority for Science and Education SASE (Rosobrnadzor).
Katerina Guba
Photos: Who are the SASE’s experts, who are assessing the quality of education in Russian universities? (Free photobank, CC BY-NC-SA 2.0, bit.ly/2z1KMG5)
The idea of university autonomy is based on the open opinions of scientists on a number of issues. One of these issues is how to assess the quality of education at universities.
How Are Universities Assessed?
There is a variety of formats how to assess the university’s work, but the format which is used more is when a university is assessed by experts from the university community. When it comes to the quality of studies, the experts are not needed because in this case various metrics on publications and citations can be used. When it comes to accreditation or audit procedures, experts must personally visit universities to see processes at a place. It is necessary because the audit procedure is complicated.
What is the university performance and how it can be measured? There is no single opinion on these questions. Experts become aware of multiple university processes and the information they gain must be countable. But the final opinion is left up to the people.
Who Carries out an Audit?
Often, an audit significantly affects university activities. Therefore, it is important to understand how experts are selected because their opinions influence the audit’s results and the further development of a university.
This was especially important for the Russian context in 2013-2016 when universities experienced close attention from the side of the regulatory authorities. These years were full of a great number of experts’ visits to universities with different kinds of inspections.
The Center for Institutional Analysis of Science and Education at the European University at St. Petersburg analyzed the work of Rosobrnadzor, whose experts were involved in audit and assessment activities. We started with the information taken from the official records of accredited experts and supplemented it with the data about universities’ Ph.D. defenses and publications.
Experts from Rosobrnadzor
The regulatory requirements say that almost any employee of a university can become an expert. Applicants must have a university degree and the experience of working at a university at least for 2 years (5 years if it comes to the experts for accreditation).
However, a deeper consideration of this issue shows that these two entry criteria alone are not enough to call a person “an expert”. In our opinion to be an expert, an applicant must:
- (1) do teaching and conduct own research – this provides him/her with the necessary experience to assess the quality of educational programs;
- (2) be a representative of a leading university;
- (3) demonstrate ethically impeccable behavior – they must not be seen in violations of academic ethics.
How are things in practice? Who were those people who received the right to be experts?
Administration-Experts
Our research shows that the expert positions are more often related to administrative and educational work, rather than teaching or research, for example:
- 75% of experts take leadership positions;
- among the university departments, the educational part is represented noticeably, thus, 35% of the experts are engaged in educational activities in their universities.
With this, the experts whose activities are related only to teaching and research, and not administration work, are much less common. In fact, representatives of educational departments come to check how well the paper processes are organized in other universities.
Private Universities
Teachers from private universities tend to become Rosobrnadzor’s experts more often. According to the latest data, 132 experts (19%) work in private universities, whereas in 2014 only 6.7% of all teachers worked in private universities.
If we say about experts from private universities, most of them (six at once) have come from the Eurasian Open Institute. At the same time, according to the data of our Center, it is private universities that Rosobrnadzor considers as suspicious institutions which require additional quality control.
Interestingly, the indicators showing the efficacy level of a university do not significantly reduce the possibility for private universities to be audited.
Leading Universities
We have found that experts from leading universities are less common, for example:
- only 11% of the experts work in leading universities;
-
only 21 universities (out of 45 leading ones) are represented with at least one expert;
- 15% of the experts work at “inefficient” (non-high ranked) universities.
A more detailed analysis of the materials presented in Monitoring of HE Institutions Efficacy has shown that if the efficacy indicators are taken as criteria to select experts, a number of experts from the leading universities will be a rare case.
A weak point for experts is their low rates for publications in Scopus – the database that tracks the citation of articles published by scholars in scientific journals. The accredited experts have come mainly from those universities that either do not know what international science is or how to publish a paper in high-index journals. The reasonable doubts arise how experts who do not work at the leading universities can qualitatively do the expertise.
Expert Citation Index
Experts may not represent the leading universities, but they must demonstrate consistency in their work. There are no data on the quality of teaching, but we can analyze their publication indicators.
Data analysis of RSCI (Russian Science Citation Index) has shown that on average all experts have a great number of publications, but not many of them are from the high rating journals. At the same time, among non-experts, there are more people with fewer publications, but in higher quality journals.
This model of fabricated publication rating (great number but low quality) is typical to managers of Russian universities, who, while dealing with the administrative affairs of the university, make it seem that they do scientific work.
Violation of Academic Ethics
Database Dissernet has shown that a harsh violation of academic ethics is more common among Rosobrnadzor’s experts than on average among Russian scientists.
Such being the case, the entire list of experts has not been checked. At the same time, Dissernet has found in its database 224 cases when the dissertations of the experts turned out to be under additional review. The main claim concerns incorrect references (plagiarism) – 10%. Additionally, the audit has revealed 22 “leaders” of the “dissertation industry” – here we mean people who took part in organizing more than 5 fake dissertation defenses.
The audit for scientific articles has shown that 25% of the experts used plagiarism in their publications. In total, according to the Dissernet report, 183 experts (81% of those in the sample) have any of the three forms of a violation of academic ethics. However, people, who were noticed in violation of academic ethics, not only continue to work as experts in university audits, but also are the labour strikers involved in dozens of audits per year.
The results of the study, conducted by the Center for Institutional Analysis of Science and Education, call into question the data about the quality of universities that were obtained during the campaign aimed at “cleansing universities”.
The very idea of audits is based on the peer-professionals assessment. If we cannot call an expert assessing the universities a high-level professional, then the audit itself cannot be recognized as valid, even if it has been carried out correctly.
The absence of serious selection requirements increases the risk that “pocket” experts of Rosobrnadzor will become more and more and this will mean that the verdict they have made will not describe a true picture and cannot be an independent opinion. Several cases of the audits for the leading Russian universities (the European University at St. Petersburg and Shaninka – the best in their fields) have shown that the doubts in the independent assessment are reasoned.
How to Improve Assessment of the Quality of Education?
In response to criticism from the university community (including an audit procedure), Rozobrnadzor has agreed that significant changes are needed. Recently Rozobrnadzor has proposed the tightened requirements to the experts engaged in university audits. According to the draft, the experts must have publications in leading national and/or foreign peer-reviewed scientific journals over the past 5 years (to date, such information has not been collected).
But will those who really meet the new requirements agree to become an expert and submit their applications for this to Rosobrnadzor? Russian scientists know how to fabricate indicators – they really know how to do it. Even the requirement “have publications in journals from Scopus database” is not an unachievable criterion due to the opportunity to be published in vulnerable journals and conference proceedings that are included in Scopus.
However, we think that the measure which is needed in this situation is not increasing the level of publication, not only this at least but reforming the entire system used in university audits and first, the bureaucracy must be reduced.
Now a typical procedure of any audit is a documentation check that requires many people, not to even mention time and space (rooms). It is hard to imagine that a good scientist would voluntarily agree to spend his/her time checking whether records, curricula, plans, and transcripts have been filled out correctly.
If they agree to do this, then on duty – to understand how audits are carried out and what to expect when the next time Rosobrnadzor commission arrives at their university. For a reason, 35% of the experts work in the educational departments of universities.
A real assessment system should be based on the unique knowledge of experts. Now it is pure paper work that requires the experts’ knowledge of regulations and norms, not their expertise in education and scientific fields. Only in this case, teachers will see the reason in the expertise and will perceive it as work for the benefit of the academic community.
Now “within a “regulatory guillotine” there are extensive discussions of new approaches to assessing the quality of education. A window of opportunity has been opened to ensure that the new assessment system does not repeat the mistakes critical to the effectiveness and legitimacy of the academic education system.
0 Comments