"The" Social Credit System
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 165 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39281 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
System programmingRoundness (object)Lecture/Conference
00:48
Translation (relic)BitHypermediaStreaming mediaSystem programmingMeeting/Interview
01:54
MathematicsMachine learningSystem programmingComplex (psychology)Observational studySemiconductor memoryWordDifferent (Kate Ryan album)Observational studyPoint (geometry)Machine learningTouch typingLibrary catalogSystem programmingUniverse (mathematics)View (database)Virtual machineAddress spaceEmailSurfaceComputer animationMeeting/Interview
04:35
HypermediaSystem programmingControl flowSystem programmingOcean currentHypermediaState of matterDegree (graph theory)Video gameComputer animation
05:38
System programmingPerspective (visual)Sound effectIndependence (probability theory)Video gameWave packetGoodness of fitDifferent (Kate Ryan album)Computer animationLecture/Conference
06:35
System programmingImplementationNumberCellular automatonDatabaseThermal conductivitySystem programmingSimilarity (geometry)Personal identification number (Denmark)Computer animation
07:43
System programmingLine (geometry)AdditionMereologyVideo gameElectronic mailing listUtility softwareDifferent (Kate Ryan album)Database
10:05
System programmingGroup actionImplementationQuicksortCategory of beingSystem callLecture/Conference
11:12
outputMaxima and minimaQuicksortInformationVideo gameCalculus of variationsSorting algorithmoutputVideoconferencingSystem programmingUtility softwareTwitterXMLDiagramProgram flowchart
12:24
Menu (computing)AlgorithmDrum memoryInformation securityWechselseitige InformationRange (statistics)State of matterSystem programmingSimilarity (geometry)Distribution (mathematics)Electric currentResource allocationAdditionCategory of beingService (economics)Wave packetSystem programmingVery-high-bit-rate digital subscriber linePlanningPay televisionInternet service providerDistribution (mathematics)INTEGRALOcean currentServer (computing)Range (statistics)Wechselseitige InformationResource allocationInformation securityProgram flowchartComputer animation
14:31
QuicksortMultiplication signMedical imagingPoint (geometry)System programmingFirewall (computing)Meeting/Interview
16:14
Formal grammarSubstitute goodComputer networkSystem programmingLocal GroupInformationResource allocationSubstitute goodSystem programmingHierarchyResource allocationFormal grammarSoftwareGroup actionRight angleRegular graphComputer animation
17:13
System programmingInheritance (object-oriented programming)Staff (military)NumberMeeting/Interview
19:06
System programmingDifferent (Kate Ryan album)outputNumbering schemeSystem programmingMoving averageDifferent (Kate Ryan album)Computer animation
19:59
Complex (psychology)Function (mathematics)System programmingoutputMathematics
20:56
Element (mathematics)Mechanism designThermal expansionContext awarenessRule of inferenceSystem programmingElectric currentLibrary catalogForm (programming)Numbering schemePlanningPoint (geometry)PlastikkarteDesign by contractUtility softwareData managementInformationService (economics)System programmingLibrary catalogFocus (optics)Point (geometry)Computer animation
21:42
Element (mathematics)Thermal expansionMechanism designContext awarenessRule of inferenceSystem programmingElectric currentLibrary catalogForm (programming)Numbering schemeCentralizer and normalizerGoodness of fitSystem programmingElement (mathematics)Thermal expansionFlow separationRule of inferenceSystem administratorState of matterMechanism designInternetworkingRegulator geneData managementHypermediaForm (programming)Computer animation
22:58
Local ringLibrary catalogSystem programmingNumbering schemeSystem programmingNetwork topologyRule of inferenceMeeting/InterviewComputer animation
24:12
Level (video gaming)Point (geometry)System programmingInformationRule of inferenceMeeting/Interview
25:09
Semiconductor memorySystem programmingPoint (geometry)FamilyDifferent (Kate Ryan album)ResultantLecture/Conference
26:22
System programmingPointer (computer programming)AlgorithmMechanism designNumbering schemeMultiplication signMachine learningRule of inferenceFunction (mathematics)Machine visionAlgorithmNeuroinformatikSystem programmingVirtual machineArtificial neural networkComputer animation
28:01
Level (video gaming)Model theorySystem programmingArithmetic meanMechanism designLecture/Conference
29:16
Latent heatElectric currentModel theoryAverageSystem programmingDistribution (mathematics)Bound stateResultantInformationLatent heatModel theoryRational numberHeuristicOcean currentComputer animation
30:53
Pairwise comparisonAxiom of choiceCharacteristic polynomialSystem programmingLocal ringSingle-precision floating-point formatPoint (geometry)Rule of inferenceMachine learningAlgorithmInformationMechanism designSingle-precision floating-point formatPoint (geometry)Local ringVirtual machineExpected valueQuicksortSystem programmingAxiom of choiceSubsetData structurePower (physics)Adaptive behaviorFamilyComputer animation
32:46
Point (geometry)System programmingBitOrder (biology)Characteristic polynomialFamilySimilarity (geometry)Virtual machineLevel (video gaming)Meeting/Interview
34:32
Integrated development environmentGreatest elementRevision controlRule of inferenceNegative numberSystem programmingRule of inferenceSimulationNegative numberCombinational logicGreatest elementRevision controlSystem programmingRight angleComputer animation
36:13
Point (geometry)Adaptive behaviorError messageSystem programmingPosition operatorService (economics)
37:49
System programmingLocal GroupSound effectSystem programmingComputer animation
38:53
Machine visionTask (computing)System programmingGroup actionPersonal digital assistantAlgorithmData structureDuality (mathematics)CASE <Informatik>PlastikkarteTask (computing)Group actionNeuroinformatikWave packetSet (mathematics)BitMachine visionSpywareSystem programmingLecture/ConferenceComputer animation
41:17
Point (geometry)System programmingData structurePlastikkartePower (physics)Mereology
42:56
LeakInformationSystem programmingInformation privacyInformation securityInformationInformation privacyInformation securityDatabaseService (economics)Wave packetComputer animation
44:08
InformationAsymmetryAcoustic shadowDefault (computer science)System programmingDefault (computer science)DeterminantAcoustic shadowInformationLecture/ConferenceComputer animation
45:02
FrequencyModel theorySystem programmingDistribution (mathematics)Point (geometry)Decision theoryMultiplication signMeasurementSystem programmingQuicksortResultantComputer animation
46:20
Default (computer science)MereologyDecision theoryDistribution (mathematics)Meeting/Interview
48:27
RankingSystem programmingModel theoryImplementationSystem programmingOrder (biology)Food energyPoint (geometry)CodeRankingQuicksortDifferent (Kate Ryan album)Model theoryNumberImplementationInformationTerm (mathematics)Computer animation
50:33
System programmingInformation privacyRaw image formatPoint (geometry)QuicksortMereologyPerfect groupSystem programmingMeeting/InterviewComputer animation
51:31
TwitterSystem programmingDependent and independent variablesVideo gameLecture/Conference
53:09
System programmingAssociative propertyArithmetic meanPressureMereologyQuicksortTerm (mathematics)Lecture/Conference
54:09
Model theoryInteractive televisionRule of inferenceArchaeological field surveyDifferent (Kate Ryan album)Level (video gaming)QuicksortMereologyAdaptive behaviorSystem programmingLecture/Conference
55:40
System programmingFrequencySeries (mathematics)QuicksortState observerProcedural programmingSimilarity (geometry)Multiplication signSpecial unitary groupInformationLecture/Conference
57:58
Lecture/ConferenceMeeting/Interview
58:47
SmartphonePoint (geometry)System programmingLecture/ConferenceMeeting/Interview
59:57
HierarchyMultiplication signExpected valueSystem programmingMeeting/Interview
01:00:47
Semiconductor memoryCartesian closed categoryInflection pointDiagram
Transcript: English(auto-generated)
00:18
All right.
00:20
Then it's my great pleasure to introduce Toni to you. She's going to talk about the social credit system, which is kind of feels to me like a Black Mirror episode coming to life. So slightly nervous and really curious what we're going to learn today. So please give a huge warm round of applause and welcome Toni.
00:42
Hey, everyone. Before I'm going to be talking and going into my talk, I'm just going to be presenting the Chinese translation streams for everyone who doesn't speak English.
01:03
So I figured it would be good to have it in Chinese as well, and I'm going to be
01:30
talking today about the social credit system in China, where the social credit system that you always hear about in Western media actually doesn't really exist. And most of my talk will actually be talking about what all we don't know, which could
01:44
fill an entire hour or even more. But I'm just going to be focusing on some of the most interesting things for me. First of all, a little bit about me. I'm an economist, but I'm not only concerned with money. I'm kind of looking at economics as the study of incentives, which means that what I'm
02:06
really interested in is how humans respond to different kinds of incentives. I don't believe that humans are completely rational, but I do believe that humans do try to maximize what they think is their best interest.
02:20
Now some words about me. I studied math, economics, and political science in a couple of different cities all around the world. I spent overall 19 months in China. Most recently I was there in July on a government scholarship, which was really, really interesting because while there I read all of these Western newspaper articles about the Chinese
02:41
social credit system. And I went to a pretty good university and I asked them, so what do you think about this system? And most of them basically looked at me bluntly and were like, what system? I haven't even heard of this. So that was kind of an interesting experience to me because in the West it's like this huge all-encompassing system.
03:01
And in China, most people that aren't directly in touch with it actually don't know anything about this. I'm broadly interested in the impact of technology on society, life, and the economy obviously. And in my free time, I do a lot of data science and machine learning with Python and R.
03:20
So I thought it was quite interesting to look at the social credit system also from this point of view because you always heard that it's like this big data initiative. And then when coming down to it, what you actually see is that they don't actually use machine learning all that much. They have basically a rule-based catalog where if you do this, you get 50 points.
03:41
If you do this, you get 50 points. And then they actually have a lot of people that are reporting on other people's behavior. I'm going to be talking about how exactly it looks later on, but I was very, very surprised after reading a lot of the Western newspaper articles that were basically, oh, this is this big dystopia Orwellian with big data working, and then you read what's actually
04:04
happening. And they have huge lists of if you jaywalk, you get 10 points detracted from you, this kind of thing. If you want to get in touch with me, you can use Twitter, but you can also use different emails, either my professional email or my personal email address that you can both see
04:23
there. If you have any thoughts on that or are interested in this a little more, I can give you more resources as well because obviously today's talk will only be scratching on the surface. So perceptions of the social credit system. One of the interesting things that I've talked about before was how in the West and in China,
04:44
the perception is completely different. So in the West, which is from FinancialTimes.com, you see this huge, overwhelming guy, and he basically puts every Chinese person under a microscope. They're all kind of hunched over, and everyone has this score attached to them,
05:01
and they seem pretty sad and like very, very Orwellian concept. Whereas in China, this is actually from a Chinese state media. And what it says is, well, we can all live in harmony with this new system and all trust each other. And interestingly, Chinese people actually believe that to some degree.
05:21
They believe that technology will fix all these current problems in society, especially because in China currently, trust is a rare commodity. And this new system will lead to more efficiency and trust and a better life. And I have a really, really interesting quote from a Western scholar that really summarizes
05:45
the Western perspective. What China is doing here is selectively breeding its population to select against the trait of critical, independent thinking. This may not be the purpose. Indeed, I doubt it's the primary purpose. But it's nevertheless the effect of giving only obedient people the social ability to
06:03
have children, not to mention successful children. This basically plays with the idea that if you have a low score currently in the cities that are already testing this system, what happens is your children can't attend good schools. What happens is you cannot take trains.
06:20
You cannot take planes. You cannot book good hotels. Your life is just very, very inconvenient. And this is by design. This is kind of the plan. The Chinese government, they say it's a little different. The idea is about changing people's conduct by ensuring they are closely associated with it.
06:44
One of the main things about this system is there isn't very much new data being generated for the system. Instead, what's happening is all the existing data that is already collected about you is basically combined into one big database for each and every person.
07:04
Buy your ID number. So in China, once you're born, you get an ID number, which is similar to a social security number in the US. We don't really have a similar concept in Germany. And it used to be that your ID number was only necessary for public, like for government stuff.
07:20
But now you need your ID number for getting a bank account. You need your ID number for buying a cell phone. Even if it's a prepaid cell phone, you still need your ID number. So all your online activity that happens with your cell phone is associated with your ID number, which means you can't really do anything anonymously because it's all going
07:41
back to your ID number. There's a couple of predecessors, some of them going actually back to the 1990s, that are supposed to be integrated into the new system. One of them or like two of them are black lists. One of them is a court blacklist. So in China, courts work a little bit differently.
08:02
They tend to like giving you fines, as they do in other countries, but they also like giving you apologies to do. So one of the things, if you do something, for example, you're a company, your food safety wasn't up to par. You have to pay a fine. But in addition to this fine, you also have to write a public apology letter in the
08:28
back of your phone again, and it was a moral failing on your part, and it won't happen again. And if you don't do that, you go on this blacklist. Similarly, if you take out a line of credit and don't pay it back within three
08:42
months, or like don't do any payments for three months, you go on this debtor's blacklist. If you're on this blacklist, which again is associated with your shen feng zhong, so your ID number, what happens is you cannot take trains, you cannot take planes, your life basically becomes very, very inconvenient.
09:01
Your children can't go to good public schools, your children can't go to private schools, your children can't go to universities. All of these issues are suddenly coming up. There is also a company database that's called Credit China, which is basically similar to the public debtor's blacklist, but it's basically a credit system, credit
09:25
score for companies, and then there's the Credit Reference Center of the People's Bank of China, which is a credit score. It was supposed to be like shufa or like the US FICO for individuals. But one of the big problems in China is that there are a lot of people that
09:44
aren't part of the formal economy. A lot of people are migrant workers. They get their money in cash. They do not have bank accounts. They do not have anything, they do not have rent or utilities or anything like this
10:00
because they live in the country, so they own their own home, which they built themselves, so they didn't even finance it. And their home isn't officially theirs because in China you can't actually own property. Instead, the government leases it to you. So there were a lot of people that were not covered in this system. And I think the last data that I had was that less than 10 percent of Chinese adult
10:26
citizens were actually in the system and had any sort of exposure to banks, which is very, very little. And that meant that people couldn't get credit because banks would only give credit to people that were in the system or people where they had some sort of handling
10:43
on whether they would be paid back. Now, the implementation details of the new system are very, very scarce. But the basic idea is that Chinese citizens are divided into trustworthy individuals and what the Chinese call trust breakers. Sometimes you have five different groups.
11:02
Sometimes you have two different groups. But in general, there's sort of this cut off above this line. It's good. And beyond this line, it's bad. This is one graphic from the Wall Street Journal that just shows some of the inputs that go into the system.
11:21
And one of the things that we see is that the inputs are crazy, crazy varied. So it is, do you pay income taxes? Do you pay your utility bills on time? Do you respect your parents? However, they measure that. Do you have a criminal record?
11:42
Do you pay for public transportation or have you been caught not paying? What about your friends? Do you retweet or use WeChat to distribute sort of information against the party, which they call reliability?
12:03
In actuality, it's not about whether it's factual. It's about whether it's against the party or not. Where do you buy and what do you buy? Apparently, if you buy diapers, it's better than if you buy video games for your score. Because, you know, if you buy video games, obviously you're not very responsible. And if you buy diapers, you have a kid, you're sort of conforming to the societal idea.
12:25
And then your score is supposed to go into all these different categories. You're supposed to have better access to social services if your score is good. You're supposed to have better access to Internet services. So in theory, the idea is that at one point, if your score is too bad,
12:43
you're not allowed to use WeChat anymore. You're not allowed to use Alibaba anymore. You can't become a government worker. You can not take planes and high speed trains. You cannot get a passport and your insurance premiums will go up.
13:03
So it's supposed to be this really, really big, overwhelming system. But in actuality, what they say their stated goals are is it's a shorthand for a broad range of efforts to improve market security
13:21
and public safety by increasing integrity and mutual trust in society. So one idea is to allocate resources more efficiently. Resource allocation in China is a pretty big problem because people grow up with 1.3 billion people, so it's always going to be scarce.
13:44
And a lot of stuff is people grow up with this idea that it's just very, very scarce. And current distribution strategies, which are mostly financially based, but also often guanxi based, don't really seem fair. For example, public transport in China is highly subsidised,
14:01
which means that the price does not reflect true scarcity. So currently, the way it works is, in theory, it's first come, first serve. In practice, there's people that are buying up all the tickets, for example, the high speed train from Shanghai to Beijing, and then selling it at a profit or selling it to certain companies
14:22
that have good ties to the government. That seems very unfair. So the new system is supposed to distribute them more fairly and more efficiently. The other thing is restoring trust in people. Perceived interpersonal trust and trust in institutions is extremely low in China. If you're from Germany, you might have heard that there's Chinese gangs
14:43
basically buying up German milk powder and selling it in China. This is actually happening because in 2008, there was a big scandal with laced milk powder. And ever since then, anyone who can afford it does not use Chinese milk powder, because they don't trust the government or the regulations, the firms enough
15:03
to buy Chinese milk powder. So they're actually importing this. And the big irony is, sometimes, this milk powder is produced in China, exported to Germany, and then exported back to China. The social credit system is then supposed to identify those that deserve the trust.
15:24
And the third point is sort of a re-education of people. The idea is they want to make people in the image that the Communist Party thinks people should be. And one additional way to the punishments and rewards this could work
15:41
is the feeling of being surveyed, because you can't do anything anonymously. You will automatically adapt your behaviour because you know someone is watching you all the time. And this is how a lot of the Chinese firewall actually works, because most people I know that are sort of more educated,
16:00
they know ways to circumvent the Chinese firewall, but they also know that they're always being watched, so they don't do that, because they're being watched, so they censor themselves. As I've said before, allocation of scarce resources so far
16:20
is mainly through financial or guanxi channels. Guanxi is basically an all-permeating network of relationships with a clear status hierarchy. So if I attend a school, everyone who also attended this school will be sort of in my guanxi network. And there's this idea that we will have a system
16:41
where we are all in-group, and in-group we trust each other, and we do favours for each other, and everyone who's outside of my immediate group I don't trust and I don't do favours for. And in some ways, the guanxi system right now is a substitute for formal institutions in China. For example, if you want a passport right now,
17:03
you can of course apply for a passport through regular channels, which might take months and months. Or you can apply for a passport through knowing someone and knowing someone, which might take only two days. Whereas in Germany, you have these very regular formal institutions. In China, they still use guanxi.
17:22
But increasingly, especially young people find that guanxi are very unfair, because a lot of these are where you went to school, which is determined by where you're born, who your parents are, and all these things. Another thing that's important to understand, because the system works through public shaming.
17:41
And in a lot of Western society, we can't really imagine that. Like, I wouldn't really care if my name was in a newspaper of someone who jaywalked, for example. It would be, oh, well, that's OK. But in China, this is actually a very, very serious thing. So saving face is very, very important in China.
18:03
And when I went to school there, I actually... We had this dormitory, and it was an all foreigners dormitory, where the staff that were responsible for the dormitory felt that foreigners were not behaving in the way they should. So their idea was to put the names,
18:24
the pictures, and the offences of the foreigners in the elevator to shame them publicly. So, for example, if you brought a person of the opposite sex to your room, they would put your name, your offence, and your room number in the elevator. And of course, this didn't work,
18:41
because for a lot of Western people, it was basically like, oh, well, I'm going to try to be there as often as possible, because this is like a badge of honour for me. And the Chinese people, they figured, well, this is really, really a shame, and I'm losing my face. She brought alcohol. So this didn't really work at all. But this is kind of the mindset that is behind a lot of these initiatives.
19:06
As I've said, there's a lot of problems with, we don't really know what's going to happen. And one of the ways that we can see what might happen is actually to look at pilot systems. China has, or like ever since the Communist Party took hold,
19:23
the Chinese government has tried a lot of policy experimentation. So whenever they try a new policy, they don't roll it out all over, but they choose different pilot cities or pilot districts. And then they choose, oh, well, this is the district where I'm going to be trying this system,
19:41
and I'm going to be trying another system in another district or city. And this is also what they did for, or what they're doing for the social credit system. Now, I have three systems that I looked at intensively for this presentation. Overall, there's about 70 that I know of.
20:01
The Suining system, Suining is a city in China, the Rangchang system, another city in China, and CSOM Credit. CSOM Credit is a commercial system from Alibaba. I assume everyone knows Alibaba. They're basically the Chinese Amazon, except they're bigger and have more users and make more money, actually.
20:23
And they have their own little system. One of the problems with this kind of system that I found when I tried modelling it was that it's a very, very complex system, and small changes in input actually changed the output significantly. So usually when they try this pilot system,
20:43
they basically have a couple of pilots, then they choose the pilot that is best, and they roll it out all over. But for this kind of thing where you have a lot of complex issues, it might not be the best way to do that. The Suining system is actually considered the predecessor of all current systems.
21:02
It had a focus on punishment, and it was quite interesting. At the beginning of the trial period, they published a catalogue of scores and consequences. Here is an example. This is basically taken from this catalogue. So if you took out bank loans and didn't repay them,
21:21
you got deducted 50 points. Everyone started with 1,000 points for this system. If you didn't pay back your credit cards, you also got deducted 50 points. If you evaded taxes, also 50 points. If you sold fake goods, 35 points were deducted.
21:42
And actually, the system was abolished, I think, in 2015, 2016, because all the Chinese state media and also a lot of internet citizens talked about how it's a Noelyan system and how it's not a good system, because it's all very centralised and everything that you do is basically recorded centrally.
22:06
But Kremers writes, Nonetheless, the Suning system already contained the embryonic forms of several elements of subsequent social credit initiatives. The notion of disproitional disincentives against rule-breaking, public naming and shaming of wrongdoers,
22:21
and most importantly, the expansion of credit mechanism outside of the market economic context, also encompassing compliance with administrative regulation and urban management rules. So one of the things that is difficult for especially German speakers is that credit in Chinese, Xinyong, means credit as in loan, but also means credit as in trust.
22:43
So the social credit system is one way of trying to conflate those two, the economic credit and the trust credit, into one big system. But the Suning system basically failed. So they adapted the system
23:02
and are now practising a new kind of system, the Rongchang system. Whenever you read a newspaper article on the social credit system in the West, most people went to Rongchang because they just received a couple of awards from the Chinese government for being so advanced at this social credit thing.
23:21
But it's very difficult to call this one system because there's actually many, many intertwined systems. There is one city-level system where city-level offences are recorded, for example, tax evasion. And there's a couple of rules. If you evade taxes, your score goes down 50.
23:41
But then if you live in one neighbourhood, your score might go up for volunteering with the elderly. If you live in another neighbourhood, your score might go up for, for example, planting some trees in your garden or backyard. So depending on your neighbourhood, your score might be different. If you work for a taxicab company, for example,
24:05
they also have their own little score system and your score might go up if you get good reviews from your passengers. Your score might go down if you don't follow traffic rules. These kinds of things.
24:24
There are designated scorekeepers at each level. So each district chooses a couple of people who are responsible for passing on the information to the next higher level. About who did what. There is supposed to be an official appeals procedure,
24:43
so whenever your score changes, you're supposed to be notified. But apparently that's not happening at this point for most people. Again, it's a system of data sharing. And one thing that they haven't really disclosed yet is what kind of data is shared. Are they only sharing the points?
25:01
So if I'm in a district and I plant some trees, does the central system get the information, person A planted some trees, or does the central system get the information, person A got five points? We don't know at this point. And it would mean something very different for how the system could be used.
25:23
But still the end result at this point is that there's one score. So you have one central score, and there's all these different smaller systems that go into this score, but at the end everyone has one central score. And currently about 85% of people are between 950 and 1,050.
25:44
So you start off with 1,000, and those are basically the normal people. And then anyone above 1,050 is considered a trustworthy person, and anyone below 1,050 is considered a trust breaker.
26:02
And as I've said before, with the naming and shaming and all these things, what you can actually see here is a billboard with the best trustworthy families in Rongchong. So these are the families that have the highest scores, for example.
26:20
CSAM Credit is a little different. It's the only system that actually uses machine learning and artificial intelligence to determine the outputs. In Rongchong, for example, they have artificial intelligence, they have computer vision for the most part, and the computer vision cameras, they decide, they try to recognize you when you jaywalk.
26:41
And then when they recognize you when jaywalking, you get a small SMS. Well, we just saw you jaywalking, your score is now dropping. But how the score develops depending on your jaywalking isn't really determined by machine learning or artificial intelligence.
27:01
Instead, it's determined by rules. You know, well, one time jaywalking deducts five points, and this is stated somewhere. CSAM Credit doesn't work like that. Instead, it uses a secret algorithm and the way I talked to some people that worked for CSAM Credit or for Alibaba,
27:22
and the way they described it was, they basically clustered people based on behavior, then gave scores to these clusters, and then afterwards did basically reverse engineered their own score using machine learning,
27:41
so that whenever something new happens, you can move to a different cluster. This CSAM Credit was actually refused accreditation as a credit score in 2017, so banks are not allowed to use the CSAM credit score for your...
28:03
to use the CSAM credit score to determine whether they give you loans or not. Because CSAM Credit is quite ingenious, obviously, Alibaba wants to keep you within their platform, so if you buy using Alibaba and using Alipay, your score goes up. If you buy using WeChat Pay, which is a competing platform,
28:24
your score goes down. This uses many of the same rewards mechanism of the official government systems, and this is just an illustration of what kind of scores you can have. Apparently, your scores can go between 350 and 850,
28:42
and in Chinese, there's basically five different levels. So 385 is trust breaker or missing trust, and then 731 is trust is exceedingly high. So one way I tried to approach this issue
29:05
was through agent-based modelling. Social credit system is individual level, but what we're really interested in, or what I'm really interested in, is actually societal-level consequences. So if everyone gets this score, what does that mean for society? Agent-based modelling works quite well for that,
29:22
because it allows us to imbue agents with some sort of rationality, but with a bounded rationality. What does bounded rationality mean? Usually in economics, people assume agents are completely rational, so they are profit maximizers, they have all the information. But in reality, agents don't have all the information,
29:41
they have a lot of issues with keeping stuff in their mind. So a lot of the time, they won't choose the best thing in the world, but they choose the best thing that they see. And bounded rationality allows us to account for this thing.
30:01
It allows us to account for heuristics and these things. And what I did is I took the propensity for specific behaviour from current state-of-the-art research, mostly from behavioural economics. For example, I looked at tax evasion, and I looked at who is likely to evade taxes in a system,
30:22
and then obviously there was some stochastic, some chance element, but the distribution that I chose is related to the current research. And I also checked that my model has similar results to the Rongchang model, which I modelled at the beginning.
30:42
So on average, 87% of my users have a score of within 10% of the original score, which is also the data that Rongchang City actually publishes. Now, for the most part, I compared design choices in two axes.
31:03
One of them was a centralised system versus a multilevel system, and a rule-based system versus a machine learning system. The centralised system is basically you have a central... You have all the information is kept centrally, and everyone in China or wherever in Rongchang
31:24
has the exact same scoring opportunities. Now, if you have a centralised system, the clear expectations were pretty good, but at the same time, the acceptance from the population was really, really low,
31:43
which they found during the Suining experiment. And there's also the problem of a single point of failure. Who decides the central catalogue? And depending on who sort of has the power, it kind of just reproduces power structures.
32:01
So because you have the central catalogue, the people that are in power centrally, they are basically deciding some sort of score mechanism that works for them very well, so that they and their family will have high scores. A multilevel system has the advantage
32:21
that local adaptation kind of works, and there's sort of many points of failure. But in my model, when I allowed locals to basically set their own rules, what happened was that they competed. So it started out being this district of Rongchang, for example,
32:42
and this district of Rongchang, they compete for the best people that they want to attract. And suddenly you have this kind of race to the bottom, where people want to move where they wouldn't be prosecuted. So they move to places where there's less cameras, for example.
33:01
At the same time, there's many points of failure, especially the way it's currently set up, with people reporting data to the next high level. And a lot of the time, what we have actually seen in Rongchang was that they reported data on people they didn't like
33:22
more than data on people they did like, or their families got better scores than people they didn't know. So it also kind of reproduced these biases. The rule-based system has the advantage that people were more prone to adapt their behaviour,
33:45
because they actually knew what they needed to do in order to adapt their behaviour. But the score didn't really correlate with the important characteristics that they actually cared about. And as opposed to in this machine learning system.
34:00
You know how in Germany we don't really know the SHUFA algorithm? And I, for example, don't exactly know what I could do in order to improve my SHUFA score. And this is a similar system in China with the CSAM credit score. A lot of people don't really... They say, well, I really want to adapt my behaviour to this score,
34:20
to improve my score. But when I tried doing that, my behaviour, my score actually got worse. And you can have different biases that I'm going to be talking about in a little bit. There's also this big problem of incentive mismatch. So the decentralised rules-based systems like Rongchang,
34:44
which is the system that I analyse the most. Why? Because I believe this is the system that we're moving towards right now. Because Rongchang won a lot of awards. So the Chinese government, the way they usually work, is they try pilots, then they choose the best couple of systems, they give them awards,
35:01
and then they roll out the system nationwide. So I assume that the system in the end will be similar to the Rongchang system. Now, one problem that I actually saw in my simulation was that you could have this possible race to the bottom.
35:22
There's also this conflict of interest in those that set the rules. Because a lot of the time, the way it works is you have your company. In your company, you, in combination with your party leaders, actually decide on the rules for the score system.
35:42
But the scores of all your employees actually determines your company's score. If you employ a lot of people with high scores, you get a better score. So you have this incentive to give out high scores and to make sure that everyone gets high scores. But at the same time, the government has an incentive for scores to be comparable.
36:03
So there's a lot of incentives mismatch. The government also has the incentive to keep false negatives down. But actually, the way the Chinese system currently works is they emphasise catching trust breakers more than rewarding trustworthy people.
36:24
So false positives for them are less important. But false positives erode the trust in the system and they lead to a lot less behavioural adaptation. I was actually able to show this using some nudging research that showed that as soon as you introduce an error probability
36:45
and you can be caught for something that you didn't do, your probability of changing your behaviour based on the score is actually lower. And in Rongchang, one of the perverse things that they're doing
37:03
is you can donate money to the party or to party-affiliated social services. And this will give you points, which is kind of an indulgence system. Which is quite interesting,
37:22
especially because a lot of these donation systems work in a way that you can donate 50,000 renminbi and you get 50 points, and then you donate another 50,000 renminbi and you get another 50 points. So you can basically donate a lot of money
37:41
and then behave however you want, and still get a good score. And the trust in other people can actually go down even more in the system, because suddenly you only trust them because of their scores,
38:00
and the current system is set up so that you can actually look up scores of everyone that you want to work with, and if they don't have a score high enough, then suddenly you don't want to work with them. Now, the trust in the legal system can also decrease, actually. Why? Because trust in the legal system in China is already low,
38:23
and a lot of the things like jaywalking, they're already illegal in China, as they are here, but no one cares. And suddenly you have this parallel system that punishes you for whatever, but why don't you just try to fix the legal system? Which would be my approach.
38:42
Suddenly, illegal activity could happen more offline, and this is one of those things that is quite interesting. In countries that we've seen that have moved towards mobile payments and away from cash, you see less robberies,
39:01
but you don't actually see less crime. Instead, you see more new types of crime. So you see more credit card fraud, you see more phone robberies, these kinds of things. And this is also where things could move in the Chinese case.
39:25
One major problem is also that this new system, I've talked a little bit about this one, but it can introduce a lot of new bias and reproduce the bias even more. So, for example, China is a country of 55 minorities.
39:43
The Han are a big majority, they have about 94% of the population. So any computer vision task, we've shown that they are really, really bad at discriminating between individuals in smaller ethnic groups.
40:02
In the US, most computer vision tasks perform worse for African Americans, they perform worse for women, because all of the training sets are male and white, and maybe Asian. In China, all of these tasks are actually performing worse for ethnic minorities, for the Uyghurs, for example.
40:26
And one way that they could try to abuse the system is to basically just, what they're also doing already in Xinjiang, is to basically just identify, oh, this is a person of the minority, well, I'm just going to go and check him or her more thoroughly.
40:47
This is actually what happens in Xinjiang. If you're in Xinjiang and you look like a Turkish person, or like from Turkmenistan, from a Turkish people, you are a lot more likely to be questioned,
41:02
you're a lot more likely to be stopped, and they ask you or require you to download spyware on your phone. And this is currently what happens, and this new kind of system can actually help you with that. I've said that it can reproduce these kind of power structures.
41:23
And now obviously we all know neutral technology doesn't really exist. But in the Chinese case, in the social credit case, they don't even pretend, they always say, well, this is neutral technology and it's all a lot better. But actually, it's the people currently in power, they decide on what gives you points and what deducts points for you.
41:46
Other problem, currently the entire system is set up in a way that it all goes together with your xuanfeng zhong, with your ID card. What if you don't have an ID card? That's foreigners for one, but it's also people in China that were born during the one child policy
42:02
and were not registered. There's quite a lot of them actually, they're not registered anywhere, and suddenly they can't do anything because they don't have a score. They can't get a phone, they can't do anything really. And part of the push with this social credit system
42:21
is to go away from cash actually. So you need to use your phone to pay, but for your phone you need your xuanfeng zhong. If you don't have a xuanfeng zhong, well, tough luck for you. And currently the system in rongzheng is set up in a way that you can check other people's scores and you can also see what they lose points for.
42:44
So you can actually choose to discriminate against people that are gay, for example, because they might have lost points for going to a gay bar, which you can lose points for. Another big issue currently is data privacy and security.
43:01
Personal data is grossly undervalued in China. If you ask a Chinese person, what do you think, how much is your data worth, they're going to be saying, what data? I don't have data. And currently the way it works is if you have someone's ID number, which is quite easy to find out,
43:21
you can actually buy access to a lot of personal information for a small fee. So you pay about 100 euros and you get all hotel bookings of the last year, you get information of who booked these hotels with them, you get information of where they stayed, you get train bookings, you get access to all of the official databases for this one person.
43:45
And for another 700 renminbi, you can actually get live location data. So you can get the data of where this person is right now, or where his or her phone is right now. But if you've ever been to China, you know that where the phone is, usually the people are in the far.
44:03
SupChina actually did an experiment where a couple of journalists tried buying that, because it's actually these kind of services are offered on WeChat pretty publicly. And you can just buy them quite easily.
44:22
So one additional thing that I looked at is, because one of the things that is quite interesting is you have this idea of credit as twofold. Credit is twofold. Trust credit, but credit is also low in credit. And what if credit institutions actually use this unified credit score
44:42
to determine credit distribution? The idea is that it's supposed to lead to reduced information asymmetry, obviously. So fewer defaults and overall more credit creation. New people are supposed to get access to credit, and they're supposed to be less shadow banking.
45:01
But what actually happens? I'm not going to be talking about how I set up the model, but just about my results. If you have this kind of score that includes credit information, but also includes morally good, or measures of being morally good, what you have is, in the beginning, about 30% more agents get access to credit.
45:21
And especially people that previously have not gotten credit access suddenly have credit access. But the problem is that this social credit score that correlates all of these different issues, it correlates only very, very weakly with repayment ability or repayment wishes.
45:41
And thus, suddenly you have all of these non-performing loans. And what we see is sort of like we have non-performing loans. Banks give out less loans because they have so many non-performing loans. And then the non-performing loans are written off, and suddenly banks give out more loans.
46:03
But you have this oscillating financial system where you give out a lot of loans, a lot of them are non-performing, then you give out a lot of loans again. And this is very, very vulnerable to crisis. If you have a real economic crisis during the time where non-performing loans are high,
46:21
then a lot of banks will actually default, which is very, very dangerous for a financial system as nationed as the Chinese one. Now, what are some possible corrections? You could create a score that basically is the same as a shufa score, so that it looks only at credit decisions,
46:41
but suddenly you lose a lot of incentives for the social credit score, if the social credit score doesn't matter for credit distribution anymore. Another thing, and this is I think the more likely one, is that you have a blacklist for people that have not repaid a loan in the past. So you basically get one freebie, and afterwards, if you didn't repay a loan in the past,
47:02
then you will not get a loan in the future. You will still be part of the social credit system, and your social credit score will still be important for all of these after-access issues, but it won't be important for access to loans anymore once you've been on this blacklist, which is probably something that the Chinese government
47:21
could go behind, but it's also more effort to take care of it. Then you have to think about, well, you can't leave them on the blacklist forever, so how long do you leave them on the blacklist? Do they have to pay back the loan, and then they get off the blacklist, or do they have to pay back the loan,
47:40
and then stay not in default for a year or for five years? There's a lot of small decisions that, in my opinion, the Chinese government hasn't really thought about up until now, because they're basically doing all these pilot studies,
48:00
and all of these regional governments are thinking of all these small things, but they're not documenting everything that they're doing. So they want to roll it out by 2020, by the way, nationwide. Once they roll it out, there's a pretty big chance, in my opinion, that they have a lot of unintended consequences, a lot of things that they haven't thought about, and that they will then have to look at.
48:24
We also... Yeah, so I believe that some sort of system is likely to come, just in terms of how much energy they've expanded into this one, and for the Chinese government, at this point, for the Party, it would be losing face if they did not include any such system,
48:41
because they've been talking about this for a while. But most likely, it would be a kind of decentralised data-sharing system, and when I ran my simulation... By the way, I will make public my code. I still need some... Basically, I used some proprietary data for my model,
49:02
and I still need the permission to publish this. Once I publish this one, I will also tweet it, and we'll put it on GitHub for everyone to play around with, if you want to. And some of these implementation details that were very important in determining model outcomes were, do we have a relative or absolute ranking?
49:22
So far, all of the systems I looked at had absolute rankings, but there is a point to be made for relative rankings. Do we have one score, where basically, if you're a Chinese person, you get one score, or do we have different sub-scores in different fields? Do we have people reporting behaviour,
49:41
or do we have automatic behaviour recording? How do you access other people's scores? How much information can you get from other people's scores? Currently, if someone is on a blacklist, for example, if you have their ID number again, you can put it into this blacklist, and then they will say, oh, this person is on this blacklist for not following this judge's order.
50:04
And then it says what kind of judge's order it was. So, most likely, it will be something like this. The idea is that the social credit system isn't only for individuals, but also for firms and for NGOs. So, what kind of roles will firms play in this system?
50:22
I haven't looked at that in detail at this point, but it will be very interesting. Another idea that Western people often talk about is do people also rank each other? Currently, that's not part of the system in China, but it might be at one point. And lastly, where does the aggregation happen?
50:41
So, I said that a lot of it is actually data sharing in China. So, what kind of data is shared? Is the raw data shared? Person A did something? Or is the aggregated data shared? Person A got this score.
51:03
At this point, most of the time, it is actually the raw data that is shared, but that also has sort of these data privacy issues, of course, that I've talked about. Okay, perfect. Now, there's 10 more minutes. Thank you for your attention. If you have questions, remarks, you can ask them now.
51:22
You can catch me up later. You can tweet to me or send me an email, whatever you're interested in. Thank you very much.
51:41
If you have a question, please go to one of our microphones. If you're watching this stream, please ask your question to IRC or Twitter. We'll also try to make sure to get to those. Let's just go ahead and start with mic one. Thank you very much for this beautiful talk. I was wondering, how did the Chinese government companies
52:02
and most of all citizens themselves respond to you doing this research? Or, let's put it differently, if you would have been in the system yourself, how would your research affect your social credit score?
52:21
There's actually two different responses that I've seen. When I talked to the government themselves, because I was there in the government scholarship, and mentioned that I'm really interested in this, they basically said, oh, well, this is just a technical system. You don't really need to be concerned with this. It's not very important.
52:41
It's just a technicality. It's just for us to make life more efficient and better for everyone. So, I assume my score would actually go down from doing this research, actually. But when I talked to a lot of people at universities, they were also very interested in my research.
53:02
And a lot of them mentioned that they didn't even know that the system existed. All right, before we go to a question from our signal angel, a request for all the people leaving the room, please do so as quietly as possible, so we can continue this Q&A. The signal angel, please.
53:21
Yeah. Janix wants to know, is the score actually influenced by association with people with a low score, meaning that is there any peer pressure to stay away from people with bad scores? The season credit score definitely is influenced by your friends' scores. The wrong score so far apparently is not influenced,
53:44
but it is definitely in the cards, and it is planned that it will be part of this. I think WeChat, which is the main platform, it's sort of like WhatsApp, exactly, it can do a lot more. WeChat is still not connected to the social credit score in the wrong term.
54:02
Once they do that, it will most likely also reflect your score. All right, let's continue with mic three. I have a question about your model. So, I'm wondering what kind of interactions are you modeling or actions? Like, what can the agents actually do?
54:21
You mentioned moving somewhere else, and what else? Okay, so the way I set up my model was, I set up a multi-level model, so I looked at different kinds of levels. I started out with basically, they can evade taxes, they can get loans and repay loans, they can choose where to live,
54:45
and they can follow traffic rules or not follow traffic rules. And because these were sort of four big issues that were mentioned in all of the different systems, so I started out with these issues and looked at what kind of behavior do I see.
55:04
I used some research that some friends of mine actually sent out surveys to people and asked them, well, you're now part of the system, did your behavior change, and how did it change depending on your responses,
55:21
depending on your score, and depending on the score system that exists. And I basically used that and some other research on nudging and on behavioral adaptation to look at how likely is it that someone would change their behavior based on the score.
55:40
All right, let's do another question from the interwebs. Yeah, it's actually two questions in one. How does this system work for Chinese people living abroad or for non-citizens that do business in China? Currently, the system does not work for non-citizens that do business in China
56:00
because it works through the Shen Fen-Zhong. You only get a Shen Fen-Zhong if you're a Chinese citizen or you live in China for 10 or more years. So everyone who is not Chinese is currently excluded. Chinese people not living in China, if they have a Shen Fen-Zhong, are on this system, but there's not a lot of information.
56:24
All right, Mic 4. Well, we've come a long way since the book series was written. Can you tell us anything about the dynamic in the time dimension? How quickly can I regain credit that was lost?
56:42
Do you have any observations there? So in the Sunning system, what they actually did was they had a very, very strict period. So if you evaded taxes, your score would be down for two years and then it would rebound. In the Rongchang system, they did not publish this kind of period.
57:02
So my assumption is that it's going to be more on a case-by-case basis, because I looked at the Chinese data, I looked at the Chinese policy documents, and they didn't really, for most of the stuff, they didn't say how long it would count. For the blacklists, which was kind of the predecessor
57:21
that we look at currently, the way it works is you stay on there until whatever the reason for the blacklist has been resolved. So you stay on there until you send off this apology that the judge ordered you to. And then, usually, you still needed to apply to get off.
57:41
So for blacklists, it does not work that you automatically get off. You need to apply, you need to show that you've done what they ask you to do, and then you can get off this blacklist. And I assume it will be a similar sort of appeals procedure for the system. All right, let's go to mic two.
58:01
Thank you. I just wanted to ask if looking up someone else's data in detail, like position, etc., does affect your own score? Currently, it apparently does not, or at least they haven't published that it does. It might in the future, but most likely, it's actually behaviour that they want.
58:21
So they want you to look up other people's scores before doing business with them. They want you to basically use this to decide who you're going to associate with. Thank you. All right, do we have another question from the internet, maybe? Yes, I do. Stand by. The question is, how is this actually implemented
58:43
for the offline rural population in China? Quite easily, not at all at this point. The idea is by 2020 that they will actually have all of this implemented. But even for the offline, or let's say offline rural population in China is getting smaller and smaller.
59:03
Even in rural villages, you have about 50-60% of people that are online. Most of them are online via smartphone, and their smartphone is connected to the Shenfeng Zhang. So it's not very complicated to do that for everyone who's online.
59:21
For everyone who's offline, of course, this is more problematic. But I think the end goal is to not have people offline at all. All right, let's jump right back to microphone two, please. Okay, thank you for the very good and frightening talk so far.
59:42
At first, I have to correct you in one point. In Germany, we have a similar system because we have this tax ID, which is set from birth on and rests 30 years after a person's death. Yeah, so we have a lifelong ID. You're right, I just...
01:00:00
I don't know mine, so I figured... No problem, but at least we could establish a similar system if we have a government which would want it. The second question from me was, you mentioned this Guanxi. Is it a kind of a social network? I didn't understand it really.
01:00:21
Yes, it is a kind of social network, but one that is a lot more based on hierarchies than it is in the West. So you have people that are above you and people that are below you, and the expectation is that, while it's a quid pro quo, people that are above you in the hierarchy will give you less than you will give to them.
01:00:44
Okay, alright. Unfortunately, we are out of time. So please give another huge applause for Tony.