How We Killed Our Most Loved Service and No One Batted an Eye
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 16 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/47584 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
6
11
12
00:00
Lecture/Conference
01:58
Computer animation
02:28
Computer animation
03:59
Computer animation
06:09
Program flowchart
06:26
Computer animation
07:26
Computer animation
07:48
Computer animation
08:10
Computer animation
09:11
Computer animation
10:13
Computer animation
10:56
Computer animation
11:29
Computer animation
12:08
Computer animation
13:00
Computer animation
13:59
Computer animation
15:29
Computer animation
15:55
Computer animation
16:33
Lecture/Conference
Transcript: English(auto-generated)
00:09
So, our next presenter is Matthias Frasterus, I hope I said that correctly, once again from the National Library of Finland. So take it away.
00:22
Thank you. Hello everyone. I have the prestigious honour of being last, so yeah, I will be telling a tale of tragedy, but one of ultimate triumph about how we killed our most loved service and no
00:42
one better than I. We actually did it twice. First we did it very wrong and that's the tragedy part and then we tried to learn from it and that's the triumph part and we hope that we can maybe share this experience and maybe someone can avoid the tragedy part.
01:06
So first a bit of background. The general Finnish thesaurus is the most used thesaurus in Finland. It is being used outside the library domain as well and it was originally developed in
01:22
the 1980s. It's monolingual. Finland has two official languages, Finnish and Swedish, but the general Finnish thesaurus is only in Finnish. There's a Swedish counterpart called Alars, which is really, really close to the Finnish one structurally.
01:43
There's very minor differences, but they are separate and the structure of the general Finnish thesaurus is very much tied to the Mark 21 format, so very much to the library world in that sense.
02:00
Then originally the thesaurus was published as printed books, but in 1999 it was decided that maybe it should be available on the web and a dedicated site for browsing it, both the Finnish and the Swedish version as well as the music domain thesauri was developed
02:25
and this site was called Vesa. It looked like this, not very pretty, maybe very 90s in design, but it was made for a
02:43
single purpose. It was very functional and I can sort of spoil the ending, we killed it at the beginning of this year, so it was used for a pretty long time and the users were very used to
03:03
it and its works and so on. It was very fast, very reliable and whenever we conducted the user survey, this Vesa was always at the top of the heap. There was absolutely nothing wrong with it and no improvements were needed, but as it
03:25
turns out sometimes the customers are wrong. There was no proper support for linked data, no APIs, you couldn't use the thesauri through the service.
03:41
And the software that it was built on was getting older and older and more difficult to maintain, so it was a maintaining that was a sort of a strain to the resources and if we wanted to do something new, we needed a new service.
04:01
So there was this research project in all the university and the University of Helsinki which began in 2003, lasted until 2012, so a nine-year project. The aim of the project was to build a semantic web infrastructure for Finland and one of
04:24
the things that were built was Onki, which was this general thesaurus and ontology service. The idea was that it should handle moderately complex all ontologies as well as the thesaurus and classifications and it would have the APIs to allow you to integrate it into your
04:44
own system and so on. And along with this, the general Finnish thesaurus, YSA, was transformed or a new version was built called YSO, general Finnish ontology, where we moved from terms to concepts,
05:03
used URIs and built a fully multilingual one vocabulary and with machine understandable semantics. The first version of Onki was built by engineers for engineers.
05:20
There was some input from the actual users, but not so much. And when it was built and as sort of a result that was probably in hindsight surprising to know whenever, the users weren't very happy with it.
05:44
So Onki 2 came about, it was a failed attempt to replicate VESA as closely as possible because that was very loved, but unfortunately it wasn't possible because what we were trying to do was much more complex than what VESA was trying to do and trying to import the
06:03
good things about VESA was basically impossible. So Onki 2 was very short-lived. And finally came Onki 3 which tried to sort of, well, tried to take some of the, well, be a compromise of sorts, take into account. Some of what the users wanted and so on.
06:23
But the main point here was that Onki was a research prototype and we weren't interested or the research group wasn't interested in usability research as such. So Onki was a proof of concept and also it wasn't very reliable, it crashed often
06:46
because reliability also wasn't very interesting as a research thing. But still Onki was slated to replace VESA in 2011 and then we got lots of feedback.
07:00
And much of it wasn't positive, so we decided to not do it and pulled back. So there was quite a bit of communication that we are going to, VESA is going to die in 2011 and two days before the date that it was supposed to die,
07:22
we pulled back, but no, no, no, bad idea. So a better idea was hatched and it was to build a production version of Onki. So in 2013, Finta project was established in the National Library of Finland, funded by the Ministry of Finance and the Ministry of Culture and Education
07:42
and the idea was to build a national level Thesaurus and ontology service. And it looks like this if you go to finta.fi. The user groups for Finta were quite a bit more than for VESA. VESA was just for the annotators while Finta tries to cater to the needs
08:05
of the vocabulary developers as well as the application developers. And we had quite a few design principles. First of all was the reliability and fast, being fast.
08:24
And also that no extra workload should be put to the annotators so that the user interface should be as easy to use as VESA or better. And we put a lot of emphasis on usability. And I will get to that in a moment.
08:43
But all of this was done in deep collaboration with people with real annotation experience. So the project included several people directly involved in this and they took active roles in the design so we could get feedback straight away.
09:03
And we also, there was some digging up information based on the VESA logs and so on. So we tried to learn as much from the failure as possible. So I want to highlight a few things. First is the user testing.
09:21
As it turns out, professional annotation or indexing work isn't very common. So the pool of test subjects that we had was pretty limited. And it was also pretty heterogeneous. So there were annotators who annotate books or museum artifacts
09:42
or health documents or whatever and they differ from one another. Then there was people who do it in Finnish, people who do it in Swedish, people who do it in both. And finally the difference between doing it as the bulk of your work versus doing something else most of the time
10:01
and then annotating as an auxiliary thing. So we needed to plan this testing quite well because the test subjects were so limited. And we needed to sort of choose the things that we wanted to test
10:22
and then choose the test subjects based on sort of that. And since the pool was unlikely to get any bigger, we had to sort of plan the testing well in advance.
10:41
And, yeah, we set up these few tasks and done a few testing rounds trying to sort of get to the bottom of usability. And in order to get the nice number, we used system usability scale
11:02
which is this lightweight tool for measuring usability. It has 10 questions which are pretty simple and you get a score from 1 to 100 with actually 68 being average based on empirical research. So 50 isn't average but 68.
11:21
But it's a nice tool in that it's easy to implement and then you get some sort of benchmark with it. So we tested things and as you can see, Onge 3 got 48 which is quite a bit lower than 68 which is the average. And Finto got, well, more than average every time we tested.
11:46
Unfortunately, we couldn't test VESA as finding a professional annotator who didn't use it or hadn't used it before. Wasn't possible, so we couldn't test it. But maybe that's good because it might have gotten 100 or something
12:07
and then we would have looked bad. Another thing that we did was very patient long-term communication. We went to all the different happenings
12:21
and tried to present this idea that VESA is going to die after all and Finto is going to be as good as that one. We tried to emphasize the fact that the user should get no extra workload and the new possibilities that were afforded by switching to Finto
12:41
including full support for multi-linguality and so on. And then we did a very long countdown to the end. We put up a banner on VESA site that VESA is going to die. And so on.
13:02
And finally, I would like to mention some of the new features that we wanted to sort of highlight over VESA's design. First of all, Finto is modular, so you can add various things to it. One neat thing is the integration into this Finna search and access service
13:25
so that if you go to a term page, if you go to a concept page in Finto, you get links to museum,
13:41
archive and library materials, that links to materials that have been annotated using that concept. And we also built a suggestion system for new concepts and changes to existing ones. Originally, the suggestions for new terms were done through email, but we wanted to sort of streamline that.
14:01
We built this simple web form that a person suggesting something could fill out. And each suggestion, it opens an issue on GitHub, so that allows for commenting the suggestions. Everyone can see what has been suggested
14:21
and then tell that I also need this one, this is a good term or isn't this the same term as that other one that is already in the vocabulary or whatever. And also, GitHub allows for powerful search tools and marking where the suggestion is going in the development process
14:43
so it's much more transparent that you can see that, okay, this has been accepted or hasn't been and so on. And aside from going to GitHub, the suggestions also go to the suggestions vocabulary in Finto
15:00
where you can use them straight away. They get a URI and if you're feeling brave, you can use them. If they get accepted to the vocabulary, the URI stays. If they do not get accepted, for example, the concept was already in the vocabulary, but with a different term or something, then we give a, is replaced by relation to the other concept,
15:24
so that if you use the URI, it will still point somewhere useful. Yeah, and this has gained incredibly good feedback, this system, which was a bit surprising
15:40
because we were sort of unsure whether GitHub is going to be popular with humanistic librarians, but apparently it is. So, yeah, and finally, yeah, the death of VESA.
16:02
It was unplugged in the beginning of this year and to zero negative feedback, which was sort of surprising. Now, I can't rule out the possibility that the reason that we got zero negative feedback was because we failed so horribly the first time, but if that isn't the reason, then I suggest
16:24
that if you ever do something like this, avoid the failure. Yeah, if you can. So, yeah, that was it. Thank you.
16:45
So, we have quite a bit of time and I assume quite a few questions, so, does anyone have... Someone must have a question about this fascinating project.
17:07
First, thank you for that epic saga. You said the URI for a concept that is suggested stays the same even though it gets accepted into the main Thedaurus.
17:23
That kind of surprised me. Do you give it like a second one, or can you elaborate on that for a bit? I mean, I don't know if you understand what my confusion is. Yeah, I'm not sure why. Okay, good.
17:41
My assumption is there's a second namespace where you put the suggestions into, and when they are accepted into the main namespace, they would get an ID in that namespace as well, so they are, if you look at the URI, you see it as part of the Thedaurus?
18:01
We use the same namespace for the suggestions as for the ones that get through. So, well, yeah, basically that means that they're... We sort of try to avoid having much information in the URI.
18:24
We try to make it as unintelligible as possible. So as to avoid people making assumptions based on the URI itself. So, yeah, it means that, well, there's a...
18:41
Our URI scheme is basically that there is a namespace and then a P and running number, so this means that there might be gaps in the local names, but you aren't really supposed to sort of make any...
19:04
Or you aren't supposed to use the URIs to sort of get any extra information, if that makes sense. Would anyone else like to ask a question while I'm on this side of the room?
19:24
I rather like this idea with handling suggestions. We face something similar in the A3 project. My question is this fin to include an editorial interface for Thedaurus editors, for the vocabulary makers, or if not, then what is used?
19:44
Yeah, it doesn't, unfortunately. The actual vocabulary work is done for the general Thesaurus. It's done inside the library system, which is unfortunate.
20:01
And for the YSO, for the general Finnish anthology, it is done in Top Red Composer. That's one part of the workflow that we are still sort of working on, that you could get the suggestions as nicely as possible
20:22
from the suggestions vocabulary into the actual vocabulary. Well, it works, but it's a bit messy, I think, currently. It would be very nice if there was a proper editor that you could simply use,
20:45
but unfortunately, the infrastructure around the library system is such that it doesn't really allow it currently. Further question?
21:00
Thank you very much for this talk. I'm a fan of Scosmos, and could you elaborate on the relationship between Scosmos and Finto?
21:23
Yeah, sure. Scosmos is the generalized browser and publication platform, and Finto is like a specific implementation of it.
21:42
It basically runs on Scosmos, but there is a few... Finto has a unique design. It looks different from the normal Scosmos design,
22:01
and also, naturally, the database or the triple stores are Finto-specific, but basically, they are very much the same in a way, in that Finto is merely a specific installation of Scosmos.
22:27
I'll ask if we have one more question. They seem to keep coming, so I don't want to cut anyone off. In that case, maybe I'll ask Adrian... Oh, ask us to... Please, once again, thank you.
22:48
And now I'll pass to Adrian to say some words.