Language learning, teaching, and testing with MERLIN

Objective: Your students assess their own L2 writing competence with relation to CEFR levels and thus better understand where they are in the language learning process.

Scenario: Choose MERLIN texts that match the CEFR level of your learners or a specific test tasks that fits thematically with the course content.

Your learners ...

• discuss strengths and weaknesses of an example text from the MERLIN corpus and compare it with the target hypothesis.
• work on a MERLIN task and compare their own productions with the MERLIN texts.
• evaluate each other's text productions with the help of the MERLIN rating grid (recommended from level B1).
  

And here's how:

1. Download a suitable ↘ test task.
2. Download the Merlin corpus and choose ↘ meta_ltext_THs (learner texts including target hypotheses).
3. Search for texts with this task [Entrance page↘ Filter texts with the file manager].

Objective: Compare your group with learners at the same or different competence levels

Scenario: Use the MERLIN texts to check ...

• what difficulties other learners at the same competence level are having
• where your learners are in comparison to learners with the same or a different L1.

And here's how:

1. Download the Merlin corpus and choose ↘ meta_ltext_THs (learner texts including target hypotheses).
2. Search in your file manager for texts at a specific level, e.g. B1, and with a specific mother tongue, e.g. Russian [↘ Entrance page↘ Filter texts with the file manager].

Learners' proficiency often comes in a profile, so that a learner might be more successful in grammar than, for example, in vocabulary. You can use MERLIN to explore different aspects of learners' communicative L2 competence, e.g. vocabulary range/control, grammatical accuracy, coherence/cohesion, at different CEFR levels.

Objective: Check incorrect use, overuse or underuse of words and structures  

Scenarios:

• Errors in the use of a particular structure appear repeatedly in your learners' texts, e.g. in Czech mít rád is being used with the infinitive of a verb instead of an accusative object. You want to examine whether this phenomenon is typical and in which contexts it occurs.
• You suggest that there is a correlation between the overuse or inappropriate use of a structure/lexical unit and the learners' mother tongue. For example, you observe the overuse of schon (already, yet) in written German texts of learners with Polish as their L1. You want to verify your assumption.

And here's how:

1. Open the ANNIS search.
2. Enter the word or lemma you are looking for ["schon"].
3. Choose the corpus [Czech, German, Italian] and click ↘ Search.
4. To search only texts written by Polish L1 speakers, enter additionally into the search field: & meta::_author_L1="Polish" .

For further explanations see Search and help.

Objective: Explore crucial aspects of language learning, such as learners' use of collocations, verbal aspect, and mood, etc. and find suitable examples for your own materials.

Scenarios:

• You want to illustrate the command of word order in the German main clause. 
• You search for examples of incorrect use of a certain structure to add usage notes to your materials or suggestions to avoid its overuse or underuse. 
• Your search for typical errors related to the use of a certain word or structure to identify practice focusses and find examples sentences.
• You want to analyze and compare texts with regard to certain phenomena of learner language.

And here's how:

1. Review in the Search and help in section 2 ↘ the list of annotated learner language features for the ones that interest you.
2. Open the ANNIS search to search for those features, e.g. G_Refl_type. Proceed as described here Search and help.
3. Now view the search results. Under ↘ full text you can explore the feature in context and copy phrases and sentences from the learner's text.

Objective / Scenario: You want to get a general overview of the text productions at a certain CEFR level.

And here's how: To get a general impression of e.g. B1 texts, sort or filter the MERLIN texts by files that received a B1 rating (fair average). Proceed as described on the ↘ Entrance page under ↘ Filter texts with the file manager.

Objective: Identify relevant milestones of the L2 or typical errors (in relation to the levels of the CEFR) and facilitate decisions about the selection and sequencing of content in the curriculum and syllabus.

Scenarios

• You want to examine which phenomena of learner language are especially relevant to the addressed CEFR level of the textbook or to the course level. You could do so by exploring MERLIN texts for phenomena that learners typically implement incorrectly at a certain CEFR level, but do apparently master at the next higher level.
• In turn, you might want to verify whether learners master a phenomenon or typical learner "problem" already from a certain level. It thus has not to be focused any longer.
• You would like to check which phenomena should be taken up again and again across different levels, e.g. by exploring which feature/error occurs at all proficiency levels in the MERLIN corpus.

And here's how:

1. Use the ANNIS search and the ↘  Query Builder to obtain information on the frequency of a feature/error for a distinct group of texts (e.g. writtten productions that received a B1 rating). Proceed as described under Search and help.
2. After launching the query (↘ Search) you can see the number of the query results below the search window, e.g. 1460 matches in 244 documents.
3. To get an impression of the frequency of phenomena within a given feature category, e.g. all features related to grammar, use the ↘  Frequency Analysis in ANNIS as described under Search and help ↘  4 Retrieve statistical information.

Most European language tests are (or claim to be) related to the CEFR. While the Council of Europe provides numerous helpful materials, there is not yet much empirical data (i.e. CEFR-related language samples) to support the test development process, especially for languages other than English (an overview of the available reference level descriptions can be found on the web pages of the Council of Euope; for English see www.englishprofile.org). 

We believe that MERLIN data ...

• can help to enhance transparency and quality in CEFR-related test construction for Czech, German, and Italian.
• can be used for benchmarking purposes.
• can foster empirically based development of assessment materials.

Furthermore, MERLIN data lends itself to the empirical validation of the CEFR scales (see MERLIN for research) and might be helpful for empirically based rating scale construction.

Objective: Create a common understanding of the CEFR levels in your institution 

Scenarios: Practise rating procedures of written L2 productions with your colleagues with the help of the MERLIN tasks and texts and compare your ratings with those of the MERLIN team.

• Extract a random sample of written tests on a specific task, e.g. "Neujahrsbrief an einen Freund schreiben" and have your colleagues re-rate the texts using the MERLIN rating grid. 
• Discuss your results and compare them to MERLIN ratings.

And here's how:

How to find learner texts for a specific task.

1. Select the appropriate task in the task overview and download it.
2. Download the Merlin corpus and choose ↘ meta_ltext_THs (learner texts including target hypotheses).
3. Search the file manager for texts with this task [↘ Entrance page↘ Filter texts with the file manager].