Towards Digitalisation in Examination and Grading: Best Practice and Challenges
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 94 | |
| Author | ||
| License | CC Attribution 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45669 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FrOSCon 20198 / 94
7
10
12
14
15
16
17
20
22
23
26
27
28
29
30
31
32
33
36
37
38
39
41
44
45
48
52
53
55
56
57
58
60
61
62
64
66
67
74
76
77
78
79
81
83
84
93
00:00
DigitizingFreewareOpen sourceDigital signalConsistencyOpen setElectronic signatureProcess (computing)Form (programming)DigitizingInformation privacyDifferent (Kate Ryan album)Internet service providerUniverse (mathematics)Regulator geneMereologyTask (computing)Order (biology)NeuroinformatikStudent's t-testService (economics)Process (computing)XMLComputer animationUML
01:31
FreewareOpen sourceDigital signalElectronic signatureProcess (computing)Blackboard systemAddressing modeFunction (mathematics)File formatSoftwareSource codeEuclidean vectorException handlingElectronic mailing listSocial classInformationProbability density functionComputing platformStudent's t-testFunctional (mathematics)Reading (process)Service (economics)Limit (category theory)AliasingFile formatUniverse (mathematics)Multiplication signXMLUML
03:16
Open setFreewareOpen sourceStaff (military)CoroutineInclusion mapLink (knot theory)FeedbackTask (computing)Context awarenessProjective planeStudent's t-testTask (computing)FeedbackProcess (computing)Multiplication signPersonal digital assistantObservational studyMaterialization (paranormal)Link (knot theory)2 (number)Computer animationXML
04:53
Level (video gaming)Task (computing)Statement (computer science)Social classNichtlineares GleichungssystemUniqueness quantificationFunction (mathematics)CodeOpen sourceFreewareComputerWordPerformance appraisalSimilarity (geometry)Mathematical modelPolygon meshLevel (video gaming)Universe (mathematics)Axiom of choiceComputer-assisted translationMultiplicationSingle-precision floating-point formatUnit testingDivisorCodeInformationNichtlineares GleichungssystemForcing (mathematics)Bookmark (World Wide Web)Order (biology)Graph coloringSweep line algorithmXML
06:36
FreewareOpen sourceArchitectureConvex hullFingerprintLaptopView (database)Student's t-testComputing platformOrder (biology)Student's t-testMobile appFeedbackCartesian coordinate systemServer (computing)Centralizer and normalizerLaptopOpen sourceCodeView (database)Cellular automatonFunctional (mathematics)Unit testingExtension (kinesiology)Projective planeFile formatCodeGradientExecution unitComputer animationProgram flowchart
08:22
Open sourceFreewareLaptopCodeVisualization (computer graphics)Cellular automatonSource codeComponent-based software engineeringKernel (computing)Interactive televisionExtension (kinesiology)Open setOpen sourceCellular automatonStudent's t-testExtension (kinesiology)Programming languageMultiplication signTask (computing)Connectivity (graph theory)Formal languageKernel (computing)Descriptive statisticsCodeCombinational logicDifferent (Kate Ryan album)LaptopVisualization (computer graphics)Software testingCodeXML
09:22
Open sourceFreewareLaptopSoftware testingCodeFeedbackExtension (kinesiology)LaptopExtension (kinesiology)Open sourceFeedbackFunctional (mathematics)Thermal expansionVirtualizationXML
09:53
FreewareOpen sourceLaptopInformation securityUsabilityOperations researchComputer fileWeb browserJava appletScripting languageConfiguration spaceGraphical user interfaceCellular automatonLaptopUsabilityInformation securityInstance (computer science)Graphical user interfaceScripting languageConfiguration spaceVideo game consoleComputer programmingLevel (video gaming)Radical (chemistry)File systemRoutingReduction of orderProgramming languageUser interfaceXML
11:45
Open setFreewareOpen sourceDigital signalLaptopResultantStudent's t-testDigitizingBlock (periodic table)Game controllerRoboticsStatisticsProbability theoryMathematicsReduction of orderTheoryMultiplication signJSONXMLUML
12:48
Reduction of orderOpen sourceFreewareDigital signalProcess (computing)Computer networkStudent's t-testBlogTimestampCodeHash functionAsynchronous Transfer ModeHash functionDigital signalAsynchronous Transfer ModeMultiplication signMereologyStudent's t-testProcess (computing)Reduction of orderPoint (geometry)Social classExpert systemTime zoneInternetworkingResultantDigitizingFeedbackSoftwareDifferent (Kate Ryan album)XMLUML
14:48
FreewareOpen sourceHost Identity ProtocolCodeHash functionOpen setStudent's t-testRevision controlPasswordHash functionFeedbackTimestampMultiplication signSoftwareLaptopXML
15:49
LaptopExtension (kinesiology)Cellular automatonStudent's t-testCodeFreewareOpen sourceCode generationHash functionView (database)Task (computing)Level of measurementPeer-to-peerRankingOpen setSummierbarkeitNumeral (linguistics)Task (computing)Functional (mathematics)Form (programming)Student's t-testVirtual machineSemiconductor memoryPoint (geometry)Descriptive statisticsNumberFeedbackView (database)Server (computing)Demo (music)LaptopProcess (computing)Computer fileExtension (kinesiology)Cellular automatonCodeMultiplicationTime zoneHash functionSoftware testingSquare numberAxiom of choiceRight angleValidity (statistics)Order (biology)Discrepancy theoryTheoryPhysical systemElectric generatorBitWeb pageConsistencyFunction (mathematics)outputUsabilityInsertion lossRankingArithmetic meanGreatest elementSoftware developerUML
22:45
FreewareOpen sourceStrömungsdrosselOpen setSquare numberSummierbarkeitGeneric programmingSoftware testingPoint (geometry)NumberCodeStudent's t-testView (database)Task (computing)Point (geometry)Software testingFunction (mathematics)BitIntrusion detection systemMereologyCellular automatonRight angleComputer configurationElectronic mailing listAxiom of choiceRevision controlSocial classDatabaseMultiplicationSquare numberExtension (kinesiology)Functional (mathematics)Form (programming)Price indexKeyboard shortcutCodeWritingSummierbarkeitExecution unitXML
29:41
NumberOpen setFreewareOpen sourceSoftwareExtension (kinesiology)PrototypeBitXMLComputer animation
30:36
FreewareOpen sourceOpen setCodeComputer programmingNumberFunction (mathematics)Computer hardwareNeuroinformatikPiPlanningServer (computing)Software testingTablet computerTouchscreenStapeldateiPartial derivativeStudent's t-testClient (computing)Point (geometry)LaptopFeedbackTask (computing)Limit (category theory)Computer scienceVirtual machineUniverse (mathematics)BitFaculty (division)CASE <Informatik>Flow separationSoftwareStatisticsMultiplication signProjective planeDirection (geometry)HypothesisXMLUML
35:42
FreewareOpen sourceUniverse (mathematics)View (database)BitTask (computing)WordElectronic mailing listComputer animationXMLUML
36:39
Phase transitionMotion blurProduct (business)FreewareOpen sourceSoftwareAddress spaceArtificial neural networkOpen setComputer programmingPrototypeFeedback1 (number)WordSoftwarePhysical systemVirtual machineMereologyStudent's t-testZoom lensAdditionXML
38:17
FreewareOpen sourceXMLComputer animation
Transcript: English(auto-generated)
00:07
You have everything in digital form. You can do plagiarism detection, and you can do scrambling of the assignments, which basically means instead of giving them tasks 1 to 10, you change the order around. And so if they look to the student next to them,
00:22
they have a completely different order, and it's much harder to cheat. So Min will now talk. No, oh, I forgot the university, of course. So the university is, of course, also a big part of the process. And first and foremost, the university has a legal certainty. And I mean, recently, with the General Data Protection
00:43
Regulation, we cannot just use any services which are offered by, for example, US companies. Even if they have their service in Europe, there's no real certainty. Then the university wants, of course, that no data leaves the university.
01:01
We need to have archivability. So exams need to be archived for up to 10 years. And the university wants, like all in all, a well-defined process because of the legal certainty. And the university needs to handle the deployment. So they need to give you the infrastructure for digital exams
01:22
because you need to provide people with computers, and you cannot just put them in a lecture room and give them a piece of paper. Yes. So now, Minh will talk about the existing solutions, which they are on the market. Yes, existing solution.
01:52
Down, down, down. The other one. The other down. There we go. So first, we were looking into the platforms on which we
02:03
can manage courses. Here at the university, we use ILEAS and LEIA. These are basically the places where courses, material, and PDF and all information about the class are stored.
02:21
The issue is that they do not usually come with auto grading functionality. They do not have a common assignment format. There's not usually support for coding assignments. And everything has to be done through an online portal, which
02:42
means it's difficult to conduct examinations because you want to limit the network access for students during this time. On auto grading, as you can see from our little list here, except for MB grader, everybody else
03:00
is closed source and with a paid service, which is not what we're looking for because we would like to incorporate our research into better automatic grading of assignments. And let's look at the research that we're conducting in the context of the project.
03:21
The goal of our research in general is for, first of all, improved education. And this can be done by automatically giving feedback with auto grading. And what this means is that instead of a student doing assignments and waiting for a week for the teaching assistant to have time to grade and return the feedback,
03:41
they can just hit the button. They get back a grade and ideally also link to the materials that will show them where the mistakes that they have. And additionally, there's also the benefits of learning at your own pace. So if you can give individual feedback,
04:01
automatically generate individual feedback, then the students are also able to maybe take a little more time or can go quickly in the curriculum and complete the study faster. And the second goal is, of course, with automating the grading process.
04:21
And you can reduce the amount of time for each of the teaching assistants and spend on grading the repetitive assignments. So normally, for example, in voice, you have 200 students. And then you have five TAs. And you have to grade how many assignments along during the semester and then the exams.
04:42
Then by automating this process, you will allow the teaching assistant and the professors to evaluate, spend on giving feedback to a more complicated task and also to actually supervise the students. And what can we auto grade? So these are the levels of assignments
05:02
that we are kind of divide our, these are how we usually divide, this is how we divide the levels of difficulties and assignments. The level one and level two are pretty self-explanatory. First one is have just multiple choice or single value.
05:23
There's no unambiguous solutions. The second level is code exercises, which can be done, simple code exercise that can be tested with simple unit tests. And this basically just return true or false. Level three and level four are a little trickier. So level three, definition, concepts, questions
05:45
that may have, may differs in the actual exact answer, but they can be tested against a general, but it's general terminologies such as definitions, equations.
06:00
And level four is the most difficult that does not have any unique answer. And an example is that level three question would be define what a cat is. And level four would be describe your, favorite type of pet. So there's no clear answer for that. So here the research at the university
06:21
is focused on the level three for short answer grading. And if you want to have more information about what the details of the research, then you can come to us afterwards and we'll be able to speak more. That can be done in an entire different talk. So in order to test the research that we do
06:41
with the short answer grading, we need a platform to test it on. And this is how we have been doing it so far. So we have, so we have, we use JupyterHub, which is the central server side application. And it's manage the Jupyter notebooks
07:03
and the Jupyter notebooks and the user access. So basically a student log into the JupyterHub server. They will see the Jupyter notebook view, which is basically a JSON based format
07:21
that allows for interactive coding and it has both text and code cells. And the teacher that log into the same JupyterHub will be able to view NB grader, which is views that have functionality for grading and giving feedback. NB grader is an extension of Jupyter notebook and all three are open source projects.
07:45
So here is a view of a Jupyter notebook. You can see here is the text cell and you can edit and preview the text cells and then you have the code cell where you can execute code. And for the NB grader view,
08:02
we have, they have the cells to give to grads here the actual answer of the students and the place also to give feedback. And you can also see here for the code segment, you can do unit tests, raise, assert and all that and this will be hidden from the student during the exam.
08:24
So we choose to use Jupyter notebook because it's a natural way to combine visualization, code and description documentation of the task in one single document. It gives the student the ability to give students the markdown text cells and code cells which can, okay so the combination of visualization,
08:44
code and documentation is done through this markdown text cells and code cells. And they are built around open source components like Tornado and CodeMirror and because of the way that the kernel is designed, you can have many different languages. It's originally built for Python
09:00
but I think the last time I checked, there's at least 150 different programming languages supported by Jupyter notebook. It's interactive, you can do all of this as you, during the examination, all the coding, testing. And it's very easy to extend because of a relatively well-documented extension API.
09:23
It's MBGrader is an extension of Jupyter notebook and it's also open source. It has some basic auto-grading functionality with the assertion that I've shown you before. It can automatically generate the feedback and the grades of the entire notebook afterwards
09:40
and however, because it's an extension, it's a little difficult to extend upon extensions to give it more functionality. And there are several challenges that come with using this Jupyter notebook and JupyterHub and MBGrader together.
10:02
So the first one is security. There's ability for Jupyter notebook to access the terminal directly so you can run commands like remove root, for example. Of course, this can be restricted with some level of security but then you can also access file system
10:22
and this is rather insecure. And you can do the same thing through programming language, through, for example, in Python, you can do OS import path and all that and the other problem is that even if you hide the buttons as a way to send command with a JavaScript console and this kind of inspires to maybe eventually
10:43
leads to containerize all the Jupyter notebooks or JupyterHub instances and yeah, to basically, to basically reduce the risk of the security. On the second challenge comes from usability. Usability. So the creation of assignments
11:02
not very intuitive for non-coder. Yeah, if you see before the assignments, you need to hit this buttons over here or this button to add a cell. And it's not straightforward to set up the infrastructure
11:21
of JupyterHub and Jupyter notebook and all this together. It's a lot of knowledge and configuration that you have to run and which leads to motivation for us to maybe have a user interface, graphical user interface to do this instead of open up the Veeam
11:42
or something to edit all the configuration script. And with that in mind, we will show our results and it comes back to Tim. Thanks.
12:00
So thank you. So let's look at the results. So this is the history of digital exams which we conducted in our masters courses and we started in 2012 with merely 15 exams. So 15 students taking one exam, which is here MSE,
12:20
which is mathematics for robotics and control. And then it grew over the years and last year we had this big red block, which was Scheindeskeit's theory on statistics or probability theory and statistics with 120 students taking the course. And during the semester, everything was already done using this infrastructure.
12:41
And let's look at what we gained by using actually digital examinations. And we had a serious reduction of grading time. So for this class, we had 120 students and only two graders and they submitted on Friday at 3 p.m. or they actually started on Friday at 3 p.m.
13:02
and submitted at five. And the upcoming Wednesday at 10 a.m., they had their grades and their feedback and everything was done. And if you ever studied at a university, you might know that this can take up to six weeks or sometimes even longer. So this is a huge selling point, I think,
13:20
for this kind of digital examination. And we have to keep in mind, only 50% of the questions were actually auto-graded, so the graders had to manually grade all the other questions. So another result which we got out of conducting all this is that we have a rather well-defined examination process.
13:42
So in the beginning, we generate user accounts for the exam, which are one-time accounts. We switch the whole network in a PC room to an exam network which is not connected to the internet and basically a demilitarized zone. Then the students log in with the one-time account,
14:01
solve the assignment. Then they submit and then they receive a hash code. So in the end, the hash code is a checksum of what they submitted, which makes sure that in the end, they can prove that nobody has tampered with their assignment afterwards. So when they receive a grade,
14:20
of course, if you have a written assignment, you can always go to a handwriting expert and they can tell you, okay, you wrote this part and someone else wrote this part. But with digital data, it's of course a lot different. Then afterwards, they write down this hash code for them to take home and the submissions are collected
14:41
and the network gets switched back to the normal mode. So maybe more on the hash code. So in the beginning, the users get, students get this kind of sheet. So it's the name, this is just another ID and this is the user name they get, which is the one-time user name with a simple password. So they can actually choose any place in the lecture room.
15:05
Then in the end, they will write the hash code here and the timestamp and we actually have two versions of that sheet. One will be signed by them and given to us. So they actually confirm that everything is correct and the other one is for them to take home. So afterwards, when they look into the exam,
15:23
weeks after it has been conducted, they can say, okay, this is actually my exam and I'm fine with the feedback. And this is how it would look like in a notebook later on. So you get a timestamp and you get a hash code, which is of course rather small. So it doesn't take up too much time to write it down.
15:42
All right. And what we also did is we extended the software we use. So for Jupyter Notebook, we wrote a bunch of extensions, which are all available on GitHub. So Envigrator or Jupyter Notebooks, they only have code cells or markdown cells,
16:02
which are text cells. And what we did was a new multiple choice cell type, which is easy to use, which I will show you later. It restricted view for the students during the exam because like in the beginning, the Jupyter Notebook is very powerful. You have a lot of buttons. You can change the order of cells. You can delete cells. And of course, you do not want the students
16:20
to actually get rid of the task in an exam. So we restricted everything. And we have like a toolbar where we can actually attach tests. So in the exam, the student could click on a validate button and then what they wrote gets tested, right? It will of course not be like a full test,
16:43
but more like a small test where you can say, okay, is the function name correct? Do you have the right input types? And is the output of the right type? And maybe some simple examples just to make sure that the students get some feedback already during the process. Then we also extended Envigrator.
17:04
So what we did in there was this hash code generation. And we have a new task view for grading, which is just an easier view for the graders because usually every grader gets to see the whole notebook and what we wanted to have is that every grader just sees like one single task and then can grade all the single tasks of all the users
17:24
so basically they say, okay, this grader has to do the first task and does it for all the graders, which is also important because you have consistency. So if you would have two graders grading the same task, then you will always have different grades if only slightly.
17:42
Yes, then another thing we made is this peer grading, which Minh developed. So in peer grading, the idea is that students actually evaluate themselves instead of a teacher doing that and we do ordinal peer grading, which means students get a bunch of answers, which I will show on the next page,
18:01
like this so they get like five question and five answers and they can drag them up and down to bring them in an order which they think the best one is on top and the worst one is in the bottom. So they do not even have to give points because we found out that there's much discrepancy between actual points,
18:21
which you assign to a question or an answer, but basically there's always a consensus about in which order it is. So if one answer is actually better than the other, this is easy to tell, but to tell by how much it is better is a bit harder and once a sufficient number of students
18:40
rank these answers, we get a ranking out of it and then we can say, okay, this is for the same question this answer has a score of 0.75, so this must be the best answer or at least the students think this is the best answer and we use or plan to use this system to prepare students for oral exams
19:01
because then all these questions could come up in the oral exam and they already can train with some answers and give some feedback to each other. So since this all was rather dry theory, we would like to show you a demo of how all of this works and I hope it actually does work.
19:24
So let me check this one, yes. So are we here? So this is basically the view that the grader gets
19:41
when they log into the Jupyter Hub server. So I'm not showing how to log in and everything because this machine does not have a lot of memory and you should do it like on a more powerful machine and then they have here a file view and here they have the form grader extension
20:02
which is nbgrader and what they can do is you can say, okay, make a new assignment, which let's just give one. We call it whatever FOSCON. We can give a due date or a time zone
20:20
to specify in what time zone that due date actually holds true. And then we just save it. We have this new assignment and now we need to basically make the actual assignment. So when we click on it, we get this view and we have to create a new Jupyter notebook
20:41
which is here running on Python. And in the beginning, you just see not much. So maybe we should call this, I don't know. This could be the first task. And this is a cell. So in the beginning, we can choose between code cells and text cells.
21:01
So in a text cell, I'm just gonna make this like a markdown cell where we can say, okay, task one. Please write a function to produce or that calculates squares, the square of a number.
21:24
And then we can click on it and then it gets rendered nicely. And then in this one, we could now give the solution where you just say, okay, just give the square function. And then return N squared.
21:42
Fine, and here we could then give a test where we say, okay, insert that, I don't know, square of two is actually equal to four. And this test runs fine. But we have to actually activate this cell toolbar to make sure that MBGrader knows what the cells mean.
22:03
So we go to the create assignment and then we can say this one is just read only cell because it's just a description. It gets this ID, which is randomly generated. Then you can say this one should be an auto-graded answer because we wrote a test for it. And this one should then be the auto-graded test.
22:23
Now, the problem is that you actually need to put something in here like test and then the cell ID of the one before. And so it's not very good in usability. So what we came up with is this task presets.
22:41
So we can just say, okay, I want to have a short answer task. Then we just give it a name like task one, click on okay. And automatically we get two cells, which already have the right IDs, have the right cell types and everything.
23:02
And I think I even prepared one before. No, we can also put multiple choice in there. And multiple choice is actually interesting because it looks like a markdown cell and you just give it some list options where you say, I don't know, option three. And if you render it, or just click run,
23:24
then nothing works. Well, then I think we skip this part for now because this is still prototype. But what we can also do, let's now say we made this task in a nice way. Now we can save it and reuse it later.
23:42
So we just say new task and say, it shows us which cells belong to the task. We can say, okay, this is a task that's named Square. Then we click okay. And we can see here the IDs are not really very nicely and they're actually not working because here we have the test part
24:02
and it should have the cell ID. But if we now import the task from here and give it, then we click okay. Then we can see it actually makes sure that the IDs are all correct. So this is basically an extension which we wrote which makes it a bit easier to work with it.
24:21
And in the end you could also have, let's say, a task database for one class and you could just easily put them in. Now here you would also have to give how many points this is worth. So let's say this is worth 10 points. And then we just save everything and would go back to this form-grader view.
24:43
And in the form-grader view we would now click generate and we get the student version of this assignment which we could look at. But I actually prepared one before which is a bit nicer to look at
25:00
which is here this test exam where I made like tasks like this and this test exam. So we would look at it from the student view later. You would actually generate it and then the student would log in, can pull this assignment and let's look at what the student view looks like.
25:26
So the student view, where are we? Yeah, student view looks similar but we don't have this form-grader part up here because it's just for the grader. And then the student would just open one task and then we can see all the buttons go away
25:42
so they cannot actually change the exam. You could not, you cannot even do it with all the shortcuts. And it has like a nice visual indicator which tells you, okay, this is actually a cell which you should edit and where you should do something with where your solution comes in. So this is already filled out, right?
26:02
But usually this would be completely empty. And so it would, in the beginning, it would actually look like this. There's always an indicator where it says your answer here and for students which are not familiar with markdown and how it works,
26:21
you can actually click the edit button to make sure here you added it and the preview button to preview it. Then when they can go to the next task and everything gets saved. So let's do that. If we go back, we can say everything is still saved. Let's put the correct solution in here.
26:47
And let's say here we have a coding task for the student and they have this little nice run indicator where they just write like a function. Please write a function that returns the sum of squares. And then as the next task,
27:02
we have a multiple choice task which maybe works. Yes, exactly. So this is what actually the multiple choice looks like. So you can just, it's very intuitive, you just click the right button. Then you save and then everything gets submitted.
27:21
And then we would go to the view of the teacher again. And we can see, okay, for this test exam, we already have two submissions. So let's click on that. And we see here the name of this one. Here we don't see the name. And we just click auto-grade in the beginning to run all the automated tests
27:42
so that we do not have to rate them. It gives you a log output. You do that for the other one too. Okay, and then it tells you already, so they already have a score of 10 of 30 points. And it says it still needs manual grading.
28:00
So this one would also tell you if everything is already graded because it checks, for example, if the student has not edited a cell, then it will automatically assign zero points to it. So now let's go to the manual grading part. So we click on our test exam. And we can now see, we can grade task one, two, or three.
28:22
So task two actually doesn't need a manual grade because it was the auto-graded one. So let's click to the task one. And we can see both submissions. So submission one. We can see find the closed form. We can see this is the answer. We have buttons for full credit or no credit. And this is actually the correct form.
28:42
So we give them full credit and we tell them well done, right? And this automatically gets saved in the background too. And then we can just look at the next submission and see what they did. And we can see in the next submission they didn't do anything, right? So we do not even have to grade this
29:03
because if we would look back here to task one, we can see there is no check mark for need manual grade. So it already detected. Nobody wrote anything in there. So it doesn't give any points. So we would then go to the second one which is auto-graded.
29:20
Click on it. See it and we can see here this is the answer. This is the test. The output of the test is actually appended but since there is no output because everything ran nicely, it already assigned full credit. You can still click on that but it doesn't matter. And then we can go back and look at the last task.
29:45
We can see both need manual grade. And this one does also not work yet because it's a prototype. But the rest works. So this is basically like a short overview
30:03
of the software and our modifications to it. So we've been using this for the last, let's say three years and have been starting to develop our own extensions for it last year and already have a bunch of them out
30:21
and you can all find them on our GitHub. So since this is all open source software, we like to give, of course, our extensions back to the community. And yes, it looks like we're finishing a little bit early. So then thank you for your attention
30:43
and if you have any questions, do not hesitate to ask. I'll let you go sit.
31:00
You're all afraid to pass this exam. Hi. Is there some partial grading because, for example, if you do this auto grading of the test, what should be the output of the program
31:22
does not mean that the program is written good because you can write it in one-liner or 10. So is there something detected about this? So for now, this is just testing the output. And I mean, these tests just give true or false, but you could actually adapt the test to give a number between zero and one
31:42
and then could say, okay, this is 80% correct or 50% correct but to actually check how well-written the program is, this is like a whole nother story and we're not really interested in that when it comes to these coding tasks. So it's more about that the solution is correct and then, of course, in the feedback,
32:00
we could look at it and then say, okay, this is not well-written. Maybe we deduct eight points. So you can always adapt it with manual grading. Yes. Are the students using their own laptops or do you have a PC lab with 220?
32:22
So we had problems in that we do not have that many computers and the tests needs to be done at the same time for like 200 or 300 people. Yes, so that's a good question. We actually do not allow students to bring their own hardware. We have those PC pools which are rather small.
32:41
So we have 30 places in them and what we did, we actually did it in batches. So we let them have the exam after each other when it came to this exam and we did it in, I think, five rooms at the same time. So of course, there is a limit to this and in the future, it would be very nice
33:01
if students could just bring their own hardware but it's always the question how much you can control that and how much you make them cheat with that. So we also recently got new laptops or we'll get a bunch of new laptops and I think the idea is more
33:21
that you should go towards thin clients like tablets or something which is very cheap and then everything would run on a centralized server and they would just log in and maybe get access to a virtual machine. So this is the plan and then we could even use Raspberry Pis which are stuck on the back of the screen and then I think at least the money
33:40
would not be the issue but of course, it's always an issue to find big enough rooms but that's also the case for the normal written exams because the number of students is growing and growing and we sometimes have exams here which are conducted in the Mensa, the big lecture hall, this lecture hall, all the ones around and so you have to always check
34:02
for which kind of exams you can actually apply this, yes. Yeah, I may add, we have examinations with more than 300, 400 people doing the examinations and we are trying the digital examination now with between 50 and 100.
34:21
120 was. 120 was. For Chinese statistics. So the idea is university has several PC pools, not only the faculty of computer science, also in other faculties to put them all together in a virtual network and then try to use them all simultaneously but it's a problem.
34:42
Yes, and may I add that we kind of developed it for our own master and we only have 20 to 30 students in each batch. So yes. So for us, it's a little bit easier but now the interest is growing in the university for this. Other questions? Yeah.
35:04
So I was wondering, are there feature requests by the people who are using it like bookmarking student answers or something like that? So there are a lot of feature requests by the TAs which are using NB Grader and we also have our own feature requests
35:22
but the problem is we're not quite sure if that software is going in the direction we want it to have because as Min mentioned earlier, we are trying to do this short answer grading and we're actually conducting a lot of master thesis and research projects on it and we would like to have an easy way to apply our research in there and maybe write new graders.
35:43
But yes, there's always a lot of feature requests like from other universities for this NB Grader and of course, from the TAs and professors here. So for example, this task view which I developed was a feature request where people just wanted to see one and we're trying to implement them as far as we can.
36:03
Anything quite interesting in the list of feature requests or just normal stuff? So by our professor of course, the feature request is that we actually have short answer grading so that people give a definition question and we have well not really auto grading but assisted grading where maybe already words are marked
36:22
which are important. I actually might have a picture for that which can show something which might be so this is might be a bit overcrowded but let's say you have like student answers
36:41
and then you could say, okay, please zoom in. Oh, zoom in, yes. So let's say these are student answers to question doesn't really matter what question it is and then you could say, okay, for the Grader, you mark the words which are important in green
37:01
and maybe the words which are not that good in red. So this is a prototype. So these do not really mark the nice ones but here in the middle, we have like a very long answer and it's compared to a master solution and the master solution actually says, oh, so let's first say what the question actually is. So the question is what is the prototype program
37:21
and what is its role in problem solving and someone gave a very long answer and the master solution which was given was it simulates the behavior of portions of the desired software product. So it immediately in the middle found this part and then says, okay, this is all probably very relevant
37:40
and the rest is not that relevant so as a Grader, you could focus directly on the important part and we plan to actually extend it in a way that you could then say, okay, this is actually not an important word so this word should be important to give some feedback to the autograder or the machine learning system in the back.
38:01
Thank you. Yep, thank you. Okay, any other questions? No, I don't see any more questions then I may thank you for your talk. Thank you. And thank you for your addition.