Classroom. To give you an outline of my presentation, I'm going to talk to you first about why researchers think it might be important for future and current teachers to talk about assessment and a growing new field called language assessment literacy. Then I'll define some terms and definitions in language assessment so we can be sure to

all be on the same page. Then I'll talk about some really basic principles of language assessment development before moving on to a short little history lesson on paper-based to online language assessment. Next I'll talk about developing your own online language assessment before providing you with some examples of online language assessment possibilities for the classroom.

And finally, I'll give you a little bit of a future outlook. So let's begin. Language Assessment Literacy for Professional Development According to some prominent researchers, Uta Knoch and Suzy McQueen, who wrote a book on

language assessment for professional purposes in 2020, language assessment literacy is a growing focus in the field of language assessment. It is mainly concerned with the growing knowledge and abilities required of stakeholders in the fair use of assessment.

Further researchers have suggested that language assessment literacy is a key component in teachers' professional development or even your ongoing training as you're becoming a teacher because as you know, teachers are frequently involved in the summative and formative assessment in the school setting. Both summative and formative assessment are terms I'll define for you later.

Language assessment literacy is something that Glenn Fulcher also talks about and he says that teachers have been thrust into a new world of assessment. And assessment literacy can help us navigate through this new world with the appropriate skills and knowledge. Of course, Fulcher could not have known and imagined the online world that we've been

thrust into the last two years because of the pandemic. In addition to other researchers, Benjamin Kremel and Luke Harding, in a more recent paper in 2020, argue that we need tools to measure teachers and other stakeholders' assessment

literacy. They argue, I think quite aptly, that teachers are at the front line of designers and users of language assessment so they need to be conversant and competent in the principles and practice of language assessment. I think in a way what they're trying to say is because teachers walk the assessment walk, they should be able to talk the assessment talk.

Kremel and Harding in their large-scale study sort of uncovered nine areas that they think teachers and stakeholders need to be literate in when it comes to language assessment. Of course, as much as I'd love to talk about number five, which is statistical and research methods in language assessment, I'm just going to focus on number one, which is developing

and administering language assessments. Language assessment development includes the entire process of creating to using the assessment. Of course, I'm only going to be able to touch on some of the basics because really I could spend an entire semester on this topic alone. Why should teachers really care about assessment development?

I'm going to lean on likely the most seminal work in assessment development literature in the last little while, and that's Bachman and Palmer's Language Testing in Practice book from 1996. They've also done another edition in 2010. This book is cited in many studies in language testing and assessment. And Bachman and Palmer argue that by being knowledgeable about language assessment, teachers

can be, one, more confident in their assessments and how they're developed and how they're useful for their intended purposes. Also, it increases the accountability of the assessment because then teachers can give

good reason for the grades that they're giving to students, either in conversations with a Fachleiter, so your department head, parents, and even to the learners themselves. And finally, you can really improve your students' satisfaction, both before, during, and after the assessment process.

As the students themselves will recognize the usefulness of the assessment for authentic language use in the future. So I hope that's convincing enough to move on and talk about some definitions of language assessment. I'm going to offer up some definitions and terms for language assessment.

I want to start with talking about what is language assessment. I'm going to start with a working definition, and then I'm going to reverse engineer that working definition and using some literature to try and back up why I used that working definition.

So my working definition is really the process of assigning a score to language skill, which is the learner, the language knowledge, as well as strategic competence. And it's to infer something about the learner's ability and competence to use the language

in some future use. I hope this idea is really backed up by Knoch and Suzy McQueen, who are two researchers in Australia, and they say that a test score is to infer something about the language ability of a person at a relatively near or distant future for some somewhat unknowable circumstances. This is really important for the EFL, or English as a Foreign Language, context because

we really don't know when learners will be using English in the future. Will they be using it in their companies? Will they be using it with conference calls with other people that don't have English as their first language? Or will they only be using it for traveling? So really, this definition, I think, works.

According to Bachman and Palmer, the primary purpose of language assessment is to inference and make generalizations about the use of specific language in specific domains, where the test taker is likely to use the language again in the future.

According to Fulcher and Davidson, language assessment is only useful if teachers interpret the inferences and use the results for meaningful and useful learning. I want to visualize this, and I've created a flow chart that has the sort of assessment

and the score in this grey box, which represents sort of what we call the construct in language assessment. The assessment is the physical assessment, whether that's a paper-based assessment. The score is based on the scoring model that you'll be using. So that can be a pass-fail model, that can be your percentages, that could be a letter

grade, or in German, you have the 1st, 2nd, 3rd as your notes. So that is the first part of that visualization. Then we have an interpretation of the score, and it's the teacher's responsibility to interpret the score based on whatever model they're using.

So for example, a 1 means something that's very good, and a 4 is something that's not so good. And then that's an inference. And these inferences then are broken down and based on theory. So whether you're using the competency model from Kanell and Swain from 1980, and they

talk about different literacies or competencies, whether it's sociolinguistic, strategic, linguistic or cultural competence. And Bachman and Palmer's model, theoretical model of language ability, where they talk about language knowledge and strategic competence.

And when they're talking about strategic competence, they're talking about meta-cognitive strategies like planning, goal setting, and taking action. And all of these lead into the future language use, so we're hypothesizing that a learner based on our interpretations and the inferences that we're making and the theories that we're using are able to use the language in the future.

Adding the teacher into this model, we see that the teacher is involved of course in many aspects of the assessment process. They provide the instruction, they develop the assessment, they offer up the score and interpret that score and hopefully then with feedback towards the learner, and feedback

is another aspect we'll be talking about later on in this presentation. Then they provide the informed guess, that's the inference about how close the learner has reached that goal of gaining the language ability, or becoming more communitatively competent. Okay, so I'm going to define some other terms that are important for language assessment.

The first one I'm going to talk about is stakes. So I'm not talking about the type of stake that you put on your barbecue of course, but talking about how important an assessment is for the test taker, which is usually your learner or your student, and the test user, and that's the teacher or the institute. So I'm going to start with low stakes. A low stakes test is a test that has like a classroom quiz, or a unit test, or some

sort of formative assessment. A medium stakes test is a midterm, could be finals, could be a placement test, you might consider your abitur a medium stakes test, and then we also have high stakes tests.

So high stakes tests are tests that are maybe used for immigration, education, or professional services. High stakes tests are life changing tests usually. There are large scale English language proficiency tests that we associate to high stakes, like the IELTS tests or the TOEFL test. These are tests that you take at a test center, for the most part, although things have changed

in the last couple of months with COVID. And they cost about $400. You go and bring your ID and take the test. And these give you access to international education, a lot of English speaking universities except for example the IELTS test, or you use them for immigration purposes, or access

to professional associations for jobs. Now of course, no matter what the stakes, the process of developing assessments remain the same, they're just more or less formalized. So the next two terms I'd like to define are testing and assessment.

Now I might be getting into some slightly muddy water here, because trying to define the difference between testing and assessment might be splitting hairs, and also testing and assessment are often used synonymously also in research and in literature. But I'm going to follow Glenn Fulcher's lead. He writes a paper in 2012, and he talks a little bit about the difference between testing

and assessment. So testing is often associated to the field of psychometrics. And the field of psychometrics takes a more positivist view. And the idea there is that we can measure cognitive and complex thought by using and creating instruments, and also using statistical models to measure these.

It also is often associated with large scale standardized testing. As I mentioned, IELTS is a large scale standardized test that's well known. They're norm referenced, meaning that a test taker's score is based on an average from a large group of test takers, and usually based on a sort of a normal distributed bell

curve. And often testing is considered summative, and summative is a term I'll define later in this presentation. Assessment on the other hand comes from the learning culture, stems more from the classroom, and arriving out of more social constructs. Assessment is considered often more classroom and performance based, and considered criterion

referenced, meaning that your scores are based on whether or not you've met the classroom goals or objectives or outcomes. And assessment is often considered more formative, again, a term I'll define in a minute.

You got to be careful not to get too bogged down in the terminology, however, because like I said, you can find testing and assessment used synonymously. The terms also may seem confusing because people like JD Brown define achievement test, using the word test, as tests that are designed and decisions are made with very specific

reference to a particular course. And these tests are usually based directly on course objectives. So that is sort of clearer in the area of a formative assessment. So the next two terms I want to define are summative and formative assessment.

So summative, if you think of the word summative, the word sum is in that word. So assessing the sum of a learner's abilities. This is often considered the assessment of learning. What we often associate to summative assessment or testing is that a certain period of time has passed, and you decide to assess whether or not the student or the learner has learned

what has been taught during that certain time period. So for example, a midterm exam, a final exam, maybe a larger proficiency test or your abitur could be considered a summative assessment. Formative assessment, in the word itself, formative, you see the words form or you could

use the word inform. So the assessment is used to form or inform learning and teaching. You could also say it's the assessment for learning. And examples of formative assessment include things like quizzes, class tests, maybe journal writing, worksheets, or diagnostic assessments.

So now I want to talk about some basic principles of language assessment development. Two terms that I'd like to talk about the most are validity and reliability. Any presentation about assessment wouldn't really be worth its salt if we didn't talk about validity and reliability.

So Urquhart and Weir talk about the fact that we need to be sure that we can actually test what we want to test, and that's the validity aspect. On the other hand, we also want to be able to depend on the results of our test and that they are actually providing us with trustworthy results, and that's the reliability aspect.

So to talk about validity, validity starts day one, ground zero of test development. Validity has to do with the meaningfulness and the appropriateness of the interpretations that we make based on our test results. We also need to be able to offer clear justification for our results and the interpretations of

our test scores. Also Knochen McQueen talk about being able to communicate clearly what our test intends to or maybe doesn't intend to assess. So there are two things, two questions that we can try to answer when we're talking about

validity is to what extent can we generalize our assessment scores to the entire target language use domain or area of communicative competence? The other question we can try and answer is does the score reflect the areas of language ability we want to measure and very little else? So it's this very little else piece that I would like to focus on for a minute because

if we measure other things rather than the language ability or the communicative competence depending on which model you're using is called construct irrelevant variance, meaning there's interference or contamination. So I just want to give you sort of a quick little metaphor and I want to use the example

of using a thermometer. So imagine you have a thermometer out on your balcony and it's a beautiful 20 degree day and it's measuring 20 degrees on your thermometer but then you bring out your coal barbecue and you put it real close to your thermometer while you're cooking up your steaks and then

you read your thermometer and it says 35 degrees. You realize at that moment that the barbecue is probably contaminating or interfering with that measurement on your thermometer and that's interference or construct irrelevant variance. The barbecue is causing that contamination and this can happen in language assessment as well. So in particular with online language assessment we can have possible interference through

the use of computers while taking tests. So instead of measuring students' language abilities we might actually be measuring their computer literacy skills, their ability to type or perhaps their ability to use a drag and drop option or anything else.

So we're re-measuring a computer literacy construct instead of the language ability construct. Fulcher and Davidson really kind of conclude and say that validity is that chain of reasoning or evidence that we as teachers think the results of an assessment mean and the actions that we take based on these results.

For example, are we passing, are we failing, are we recommending, are we placing, are we maybe just using the assessment for encouragement to motivate or do we want to try assess again. That is what are the consequences of our results. So if we take a look at our visual again we can see that red line running through all

these parts from the assessment down to the language use. We want to be able to use that reasoning and evidence to make valid decisions about what the assessment intends to say. So reliability. Reliability is what we call measurement consistency. So there are two types of reliability, external and internal.

So there are two types of external reliability that I want to talk about. One is test-retest. So if we were to give the test again within a certain period of time to the same test taker would we get a same or similar result. That is called test-retest reliability. The other type of external reliability is inter-rater reliability.

So would the student get the same score irrespective of who is marking or rating the test. We also have internal reliability. And internal consistency or reliability is what we usually test statistically with Chromebox Alpha or Gutmann's Lambda. This is when we do a pairwise correlation between the items within a test to see if

there is internal consistency. Are we actually testing the same construct throughout the test. A measure of one is perfect correlation which we never get. But a measure of 0.7 or 0.8 is moderate to good and can be used for classroom assessments. Naturally measuring reliability in this way requires some training and statistics and

some simple statistic analysis programming. However this can mostly be done with also spreadsheet software. Reliability of an assessment is really about the trustworthiness of the interpretation of our scores.

The last thing I want to talk about is usefulness. And usefulness is something that Bachman and Palmer really focus on in their book Language Testing and Practice. Any language test really must be developed with a specific purpose, a specific group of test takers and a specific use domain.

So those are three terms that I have used throughout this presentation which is purpose, audience and also use. So is the situation or context in which the test takers will be using the language outside the test, do you have that in mind while you're creating the test? If we take a look at the visual again we see that the language use domain or target language

use domain is at the very top. And this is what we use when we consider developing assessment and curriculum for that matter. And this is what we make decisions on and decide on what's worthwhile to teach and test. And what's useful for learners in the future. For instance you wouldn't use an engineering vocabulary word list and they actually do

exist with a group of grade nine students in Germany. These decisions however are often made for you whether it's through curriculum development at a district level or even a school level on which course books you're going to use.

Of course in terms of assessment what we want to be able to do is align our target language use domain with the future language use that we assume students will be engaging in sometime in the future.