Bestand wählen
Merken

Algorithmic science evaluation and power structure: the discourse on strategic citation and 'citation cartels'

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
try it and
and
and please give a warm welcome here I don't it's so on France's days on end units Judith's that yeah the stage thank you thank you thank you thank you and on the social and the the a kind of the
and we believe that scientific performance indicators are widely applied to inform funding decisions and to determine the availability of career opportunities that those of you who were working in science or have had a look into the science system might agree to that and we want
to understand evaluated the the metrics as algorithmic science evaluation instruments to highlight some things that that do have and that hat that do occur also with other other algorithmic instruments of some evaluation and so we're going to start
with a pair with the quality air from publication in 2015 which there's reads at the tiring after the metrics tightens its grip is having a disastrous effect on the model assigns presented to young researchers so we have heard the talk of handle already and his uh basically and talking about and also about also talking about and problems in their science system and the reputation as by the end of the indicators and the the question is is the bad guy here so it if you speak of time many of the metrics to the actor doing this or are maybe the limitations the problem and we want to contextualize are talk to the growing movement of reflexive metrics so those who are doing science studies social science social studies of science scientometrics and informetrics the movement of reflexive metrics so the to basic idea to say OK we have checked accept
accountability if we do the metrics in fact metrics we have to understand the effects of algorithmic evaluation of science and we will try not to do do not to be the bad guy and there and here the main mediator it off the am find evaluation which is perceived by the researchers is in the algorithm so I will hand over the microphone to on I will not hand over the microphone that I would hand over the top to Theresa she's going to talk about
unification of scientific evaluation OK I hope you can hear me don't yes but I have to say all right
when you think about this time system what do exist we expect what can society expect from a scientific system in general we would say a reliable and truthful knowledge that is scrutinized by the scientific community so where can we find this knowledge normally in publications cell With these publications can react to say whether science is bad or good or is there bad science and others and in the air out of digital digital publication databases there's a big datasets of publications and these are used to evaluate and calculated and the quality of scientific output so in general with this made at a and we can would tell you who is the author of a publication where the home institution of this author or a which types of citations signed and bibliographic that information so this is used as in the calculation of the parametric indicators have for example if you take the Journal uh citation at that the tonal impact factors which is a citation-based indicator you can compare different journals and maybe perhaps say which journals are performing better than others or if the journal factor has increased or decreased over the years another example would be the Hirsch index and for individual scientists which is also widely used that when scientists apply force for jobs so they put this number is in this and supposedly this tells you something about the quality of research those scientists are conducting so with
the availability of the data and we can see an increase in its you search and so in a scientific environment in which data-driven science is established a scientific conduct decisions regarding hiring of funding heavily rely on these indicators and so there is maybe a little naive believes that these indicators so that our data addresses and rely on data that is collected in the database is a more objective and metric that we can use cell and here's an a quote by reader and Simon and in this brave new world trust no longer resides in the integrity of individual truth tellers while the veracity of prestigious institutions but is placed in highly formalized procedures and acted through a disciplined self-restraint nonetheless seems to be a supplements so we see a change of and Neville agent system that is relying on expert knowledge to a system of algorithmic size evaluation In this change there's the belief and that the personalization of the system and the perception of algorithms as the role of law so all
we're looking at the interaction between the algorithm and scientists we can tell that this relationship is not as easy as it seems algorithms are not in fact objective they carry it social meaning and human agency in but they are used to construct a realities and algorithms don't come naturally they don't grow on trees and can be picked up by scientists and people of their weight of the scientific system so we have to be re refractive and think about which socially meanings the algorithm halts so when there is a college that is used in that that the algorithm uses there is a subjective meaning in this call and there's agency in this cult and you can't just say all this is a perfect construction of of the reality of scientific systems all have the belief that this tells you more about the quality of research it's not so a good indicator of so when you think about the example of citation counts the algorithm reads the bibliographic information of a publication from the database and self scientists they cite papers that relate to their studies but we don't actually know all about which factors which of these citations are more meaningful than others so they're not as easily comparable but the algorithms give you don't believe they are so relevance is not as easily have put into an algorithm and there's different types of citations so all the scientists perceived this use of the algorithms also as a powerful instrument and so the algorithm has some sway above the scientists because they rely so much on those indicators to further their careers to get a promotion or I get funding for the next research projects so all we have a reciprocal relationship between the algorithm and the scientists and this creates a new and construction of reality so we can conclude that
governance by algorithms lead to behave well adaptation and scientists and that 1 of these examples that uses the Science Citation Index as that will be given the on Francis gaps in thank to may have
the answer is likely starts a fund focusing on reputation authorship to conceive of light on 1st let me start with they talk about by Eugene
Garfield which says is it reasonable to assume that if I cited paper that I would probably be interested in based papers which subsequently cited as well as my own paper in peter I have observed on several occasions that people prefer to slightly up the articles I had cited rather than cite me it would seem to me that this is the basis for the building up of the logical network was tight excitation in services so actually this sign citation in index which is described here was mainly developed in order to solve the problem solver information retrieval Eugene Garfield also found of sitation index short SCI narrative it's all began to evaluate a huge interest in reciprocal publication behavior you recognize the increasing of interest as a strategic instrument to exploit intellectual property and indeed the interest in the SCI understated successively the kite more relevant within the discipline and its use it's accepted later price to solar social scientists asked all right for better research on the topic as it's currently also uh meta-crisis in unstated if a paper was cited wants it would get cited again and again so the main problem was that the rich will get richer which is also known as the gravity finally the SCI and its use turned into system which was and still is used as a reciprocal visitation system became a central in global actor once a paper was cited that the probability ability it was cited again was higher I did would even extend its own influence on a certain topic within the scientific field said the was now that there that you would by the read a certain article and it would say well you know people with 2 and research of asserted a topical subject so this phenomenon that which rise to an instrument of this disciplining lines and created the power structures I let me show you 1 example which is closely connected to this phenomena by just told you about and I don't I I find it here in this room that the and so the astronomers so all physicists on the enough there's here again that's great actually so the next slides
here we have a table with a window from frontier a time window from 2010 to 2060 on and social scientists from the late found that that the CNO authorship within the field of of physics and extended by 58 on a yearly basis at this time with said this is actually already very high but they also found another very extreme cases they found 1 paper which had run about 7 thousand words yeah and and the mentions all the shit of 5 thousand so you would an average of the contribution of each arm scientists the research of this type of this paper was mentioned a contributed was 1 . 1 word of strangers is a of course you have to see this if there is a certain context and maybe we can talk about this later on because it has to do with access a particle detector which uh requires hyperplane in stuff but still so the number of all the shape and you can see this regardless of which of scientific field we're talking about generally increased the last years Adams and the so it remains a problem aren't especially full of the reputation obviously it remains a problem that there were uh is such high pressure on nowadays researchers still of course we have uh athletes and research requires that uh of responsibility and for example there's
1 several ones but there's 1 and here on this slide the Australian court for the responsible conduct of research with which it says they were the right to authorship is not tied to 0 profession that does not depend on whether the contribution was paid for all monetary it is not enough to have provided material overeat Rudy technical support also have made the measurements on which the publication is based a substantial intellectual involvement is required so yeah this is 51 rule to work to work with what to work by the following I have and still we have the rule of this this problem of reputation which remains and where had over to to again thank you so we're going to stay there
to teach excitation now so if you put this on the point of reputation that are like
that if you may say so the researcher and to find something in this research and his or her research and addresses the publication and describing it to the community and the community scientific the community rewards the researcher with reputation and now the algorithm which is like perceived to be a new thing down is mediating the
visibility of the researchers and results to the community and is also mediating the reward so the career opportunities of the funding decisions and so on and what happens now art and what is possible to happen is that the researcher addresses isn't really a research also to the algorithm in terms of citing those who I am evaluated by the algorithm used and he wants to support and also in terms of like keywords has strategic keywording and so on and that's the only thing which happens new make might be a
perspective on that so the 1 thing you the algorithm is addressed as recipient of scientific publications and is
like a far-fetched discriminated so-called inclusive colleges and citation cartels am but dimming by that so invisible colleges as a term to say OK people as fighting each other and they do not work together in the coworking space maybe but they've written do research on the same topic and that's only plausible that they cite each other and if we look at citation networks and find people fighting each other that does not have necessarily to be something that and we also have people are more concerned that there might be lake citation Caswell so researchers fighting each other not for it and purposes like the research topics are closely connected but to support each other in their out like career prospects and so people do trying to discriminate those invisible colleges from citation cartels ex post from looking at metadata networks of publication and find that a problem and there we have a discourse on that ended the metrics community so and I will show you some short and yeah quotes what people like but yeah we talk about how people talk about their the citation
cartels so for example there is in 2012 said Judge Frank want us on the possibility of citation college health groups of editors and journals working together for mutual benefit so we have heard of uh about the general impact factors so they have and it's beliefs and that editors and talk to each other behaviors side might and I say to journal and we both will boost our impact factors so we have some people trying to detect those cartels and examined in and on uh as you wrote that we have little knowledge about the phenomenon itself and about where to draw the line between acceptable and unacceptable behavior so we're having no like moral discussions so about recent research ethics and also we find and discussions about the fairness of the impact factor so young it as road disingenuously manipulating impact factor the significant way to harm the fairness of the impact factor and that the very interesting thing I think because l why should be an indicator of the fear so the to believe that that we have a fair measurement of scientific quality relevance and rigor in in 1 single bound like numbers like adrenaline picked factor is not a small thing to thing to say and also we have a detector a call for detection and punishment so uh there's also wrote if disciplinary norms and decorum cannot keep this kind of behavior bait the threat of being de-listed from the data are may be necessary and so we find that the the moral concerns of
right and wrong we find the location
of the fairness of indicators and the frame the
call for detection and punishment and so when I 1st heard about that phenomenon of citation card was which is believed to exist I had something in mind which sounded but it is solid like familiar to me because we have a a similar information retrieval from discourse or and at a discourse about ranking and power in a difference in the area of society and search engine optimization so I found because that
page and I know who developed the good and the the page rank algorithm so the good Google's ranking algorithm in a 1999 which says changed and then a lot but and they wrote also a paper about the social and implications of their information retrieval by those indicators and had page rank as an indicator and road that these ties of personalized Rank of virtually immune to mean manipulation by commercial interests for example fast updating of documents is a very desirable features but is abused by people who want to manipulate the results of the search engine and that was an important reason to read because we also have like and narration of abuse of manipulation be armed perception that that that might be fear so we have a fair indicator and people try to be traded and then we had in the early 2000 Rousseau I recall having out a private website with the public guest and getting links them from people who wanted to do is their Google page rank and and shortly after Google decided to punish link spam in their ranking algorithm and then I got a lot of e-mails of people attaining please read my post from your against that because people is going to punish me that and you know we we may say that this search engine optimization discussion is now somehow settled and 6 and accept that
Google's uh ranking is useful and they they have the secret algorithm but it it works and that's widely used and although the journal impact factors seems to be and transparent and its and basically that the same thing that it's accepted to be useful and thus it's widely used so the Genentech expected the SCI and the light and we have another hour
analogies so that Google these types with decides which you behavior is regarded acceptable and punishes those who act against the rules and thus hold enormous amount of power so our which has a lot of lots of implication and led to the and spreading of content management systems for example with a search engine optimization lichens and so on and so we also have this I'm like power concentration in the hands of clarity and
from a Thomson Reuters to and the host the database as well for the journal impact factor and they decide on who who's going to be indexed and that those that of federation records and how is the algorithm in detail implemented in their their database so we we have this and power concentration their own 2 and I think if we think about this analogy we might come to interesting on the air thought that and so our time's running out so we we're going to give a take home
message and tilling so we
find that the scientific community reacts with codes of conduct to a problem which is believed to exist their strategic citation we
have a database providers which react with sanctions so that people the listed from the Journal Citation written records of our genomes are reduced from the general setting records to punish them for a citation stacking and we have researchers and
publishers to adapt their publication strategies in reaction to these and perceived algorithmic a power that some if we 1 to understand this as a problem we don't have to we don't have to only erected the algorithm that we
have to address the power structures so who hold these instruments in use in their hands if we talk about in the the metrics as an instrument and we should not only be and the other is
so has ticked of any algorithm thank you very much but few have that you functions of these and you will in reverse order of the the thank you for shining a light on all sciences actually see in units obligations and as I started off as well as more about scratching each other a little bit of some questions here from the obvious is microphone to please you think you are almost always interesting part of I have a question you may be familiar with the term measurement assumption that if you provide a worker with an incentive to do a good job based on some kind of metric then the work will start optimizing photometric instead of trying to do a good job and this is kind of so when you don't assume that made it could be treating the symptoms before you just read about the code of conduct freaking algorithms for addressing policy factors but is that we need to remove the incentives that lead to this measurement this function so and you refer to the so I would refer to this as a phenomenon as like perverse learning so learning for the year of you get but not for your intrinsic motivation to learn something and so on that's and yet we we of that observes that an and the affine system but if we only adapt the algorithm so take away the ear and centers would be like the yeah do not you wouldn't want to evaluates research at all which chicken probably yeah what we want to do but added to whom would you address this on this them called my this demand so please do not sit around have indicators on scientific that question vector for vote date questions from the audience out there on the internet please you might is not working but did I go to my phone number 1 Lisa but yeah uh water for work out of uh this but I think the fundamental problem is not all these things I think of the fundamental problems that the rethink the impact factor is a useful measurement from the quality of science because I think it's just not this and have it was also used this yeah have I would not say that the journal and take pictures of measurement of scientific quality because and no 1 was like and a definition of the scientific quality so what I cannot observe only people believe this general impact factor to reflect some quality and maybe they are chasing a ghost but I think it right whether that's the gap valid measure is not so important to me even if it were an irrelevant or in a valid measure it would concern me how it it's uh effect science a digression that from my commentary here these 2 6 was an interesting talk of the question about the 5 cells in also papers were the same they're both published 5 solvent times or was it 1 of the most and page title there was at 1 them I but have a have a accounting is more than 7 thousand words and the all authorship so old there's a cold this will more than 5 thousand doesn't the moon is roots of use is a free group room all and that's what I want that's what I had uh earlier when a society you have to see this uh within its context say of the emphasis is on a working with this uh without loss of this detective system as their words so this is in the audience they probably do them how this works by do not and but as they act like it's so much work to get to work with this I as a set route requires so hibernate it's it's of maybe obviously have the yeah uh so everyone including use was List of exactly because it that's it and if this is absolutely correct so not well this is something which is the to be discussed right this is why we have this talk as we want to make this transparent to the the contributed to an open discussion but in a I'm sorry guys I have to cover all of your because our emission of in space is so coming to ring and I suggest that you guys find a Java somewhere I made the house or so we're we're right yeah we're allowance I would love to have a was applause for these ladies from really really electrolytes on how you see how do we use more hardworking catering to go to the
i th and the
it kind the people but but
but but
Algorithmus
Einheit <Mathematik>
Datenstruktur
Leistungsbewertung
Mini-Disc
Algorithmus
Algorithmus
Linienelement
Leistungsbewertung
p-V-Diagramm
Information
Physikalisches System
Indexberechnung
Biostatistik
Entscheidungstheorie
Leistungsbewertung
Entscheidungstheorie
Beobachtungsstudie
Soundverarbeitung
Algorithmus
Linienelement
Informetrie
Leistungsbewertung
Linienelement
Soundverarbeitung
Datenmodell
Reflexiver Raum
Physikalisches System
Informationsmodellierung
Algorithmus
Inverser Limes
Indexberechnung
Reflexiver Raum
Biostatistik
Leistungsbewertung
Autorisierung
Leistungsbewertung
Linienelement
Zellularer Automat
Zahlenbereich
Physikalisches System
Rechnen
Rechenbuch
Teilbarkeit
Datenhaltung
Physikalisches System
Erwartungswert
Funktion <Mathematik>
Forcing
Prozess <Informatik>
Rechter Winkel
Automatische Indexierung
Digitalisierer
Datentyp
Indexberechnung
Information
Meta-Tag
Funktion <Mathematik>
Leistungsbewertung
Subtraktion
Gewicht <Mathematik>
Leistungsbewertung
Mathematisierung
Adressraum
Interaktives Fernsehen
Zellularer Automat
Gesetz <Physik>
Netzwerktopologie
Physikalisches System
Algorithmus
Geschlossenes System
Konsistenz <Informatik>
Datentyp
Indexberechnung
Leistungsbewertung
Beobachtungsstudie
Expertensystem
Konstruktor <Informatik>
Datenhaltung
Systemaufruf
Physikalisches System
Algorithmische Programmiersprache
Teilbarkeit
Entscheidungstheorie
Integral
Objekt <Kategorie>
Arithmetisches Mittel
Zahlenbereich
Projektive Ebene
Messprozess
Information
Programmierumgebung
Wärmeleitfähigkeit
Algorithmus
Algorithmus
Automatische Indexierung
Leistungsbewertung
Anpassung <Mathematik>
Information Retrieval
Gravitation
Mathematische Logik
Physiker
Physikalismus
Zahlenbereich
Gebäude <Mathematik>
Mittelwert
Vorzeichen <Mathematik>
Datennetz
Endogene Variable
Hyperebene
Bildschirmfenster
Datentyp
Datenstruktur
Gerade
Basisvektor
Leistung <Physik>
Shape <Informatik>
Datennetz
Kategorie <Mathematik>
Detektion
Physikalisches System
Kontextbezogenes System
Intelligentes Netz
Rechenschieber
Druckverlauf
Datenfeld
Automatische Indexierung
Basisvektor
Ablöseblase
Wort <Informatik>
Ordnung <Mathematik>
Lesen <Datenverarbeitung>
Tabelle <Informatik>
Rechenschieber
Punkt
Rechter Winkel
Code
Endogene Variable
Ablöseblase
Schlussregel
Messprozess
Einflussgröße
Wärmeleitfähigkeit
Eins
Resultante
Algorithmus
Adressraum
Adressraum
Term
Entscheidungstheorie
Algorithmus
Metadaten
Algorithmus
Datennetz
Linienelement
Perspektive
Datennetz
Inklusion <Mathematik>
Term
Raum-Zeit
Teilmenge
Bruchrechnung
Texteditor
Teilbarkeit
Gruppentheorie
Gruppenkeim
Systemaufruf
Zahlenbereich
URL
Indexberechnung
Normalvektor
Gerade
Einflussgröße
Teilbarkeit
Normalvektor
Information Retrieval
Subtraktion
Rahmenproblem
Minimierung
p-V-Diagramm
Systemaufruf
Ähnlichkeitsgeometrie
Ranking
Chipkarte
Flächeninhalt
Suchmaschine
Indexberechnung
Leistung <Physik>
Information Retrieval
Resultante
SCI <Informatik>
Web Site
Minimierung
Güte der Anpassung
p-V-Diagramm
Ranking
Binder <Informatik>
Ranking
Teilbarkeit
Homepage
Homepage
Arithmetisches Mittel
Algorithmus
Rangstatistik
Suchmaschine
Typentheorie
Indexberechnung
E-Mail
Algorithmus
SCI <Informatik>
Minimierung
Datenhaltung
Schlussregel
Content Management
Teilbarkeit
Datenhaltung
Schlussregel
Konzentrizität
Datensatz
Algorithmus
Suchmaschine
Datentyp
Analogieschluss
Leistung <Physik>
Message-Passing
Datensatz
Decodierung
Menge
Datenhaltung
Codierung
Service provider
Wärmeleitfähigkeit
Message-Passing
Datenhaltung
Message-Passing
Decodierung
Algorithmus
Datenstruktur
Linienelement
Adressraum
Strategisches Spiel
Datenstruktur
Datenhaltung
Leistung <Physik>
Abstimmung <Frequenz>
Bit
Einfügungsdämpfung
Decodierung
Wasserdampftafel
Applet
Zahlenbereich
Zellularer Automat
Term
Raum-Zeit
Code
Datenhaltung
Homepage
Internetworking
Message-Passing
Einheit <Mathematik>
Algorithmus
Prozess <Informatik>
Adressraum
Freie Gruppe
Affiner Raum
Indexberechnung
Einflussgröße
Soundverarbeitung
Lineares Funktional
Rootkit
Validität
Routing
Mailing-Liste
Physikalisches System
Vektorraum
Teilbarkeit
Datenstruktur
Mereologie
Wort <Informatik>
Ordnung <Mathematik>
Wärmeleitfähigkeit
Hypermedia
Medianwert
Systemprogrammierung

Metadaten

Formale Metadaten

Titel Algorithmic science evaluation and power structure: the discourse on strategic citation and 'citation cartels'
Serientitel 34th Chaos Communication Congress
Autor Hartstein, J.
Isigkeit, Teresa
Sörgel, Franziska
Lizenz CC-Namensnennung 4.0 International:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/34986
Herausgeber Chaos Computer Club e.V.
Erscheinungsjahr 2017
Sprache Englisch

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Quantitative science evaluation, such as university rankings, rely on man-made algorithms and man-made databases. The modelling decisions underlying this data-driven algorithmic science evaluation are, among other things, the outcome of a specific power structure in the science system. Power relations are especially visible, when negotiated during processes of boundary work. Therefore, we use the discourse on 'citation cartels', to shed light on a specific perception of fairness in the scientific system, as well as on the actors who are in charge. While doing so, we draw analogies to the discourse on search engine optimization.
Schlagwörter Science

Zugehöriges Material

Video wird in der folgenden Ressource zitiert

Ähnliche Filme

Loading...
Feedback