Submissions for WMT2023 Terminology Shared Task

Submissions for WMT2023 Terminology Shared Task

1. Folder Structure

There are two folders:

1. submissions folder contains all files in the exact form as they were submitted by the contestants;
2. split_submissions folder contains the files from the first folder that are split and re-grouped for the simplicity of evaluation (see 2.2).

the name is organized the following way: in submissions folder: {system_name}.{dataset}.{lang_pair} (e.g. DCU-TU-TR.blind.en-cs) in split_submissions folder: {mode}.{system_name}.{dataset}.{lang-pair}.tsv (e.g. rand.DCU-TU-TR.blind.en-cs)

Where:

• system_name is name of submitted system;
• dataset is in {blind, test} set and stands for the datasets used for evaluation. In the paper, we used the blind dataset unless noted otherwise;
• lang-pair is in {en-cs, de-en, zh-en} set and stands for the language pairs (ordered in translation direction).
• mode is in {base, term, rand} set and stands for the testing modes of the system. base stands for no dictionary used, term stands for the terminology dictionary-aided translastion, and rand stands for random quasi-terminology dictionary.

2. File Format

2.1. Initial Submissions

For the submitted files in submissions folder, each file is organized as following triplets of lines with input and output translations, according to the translation mode:

source sentence ||| translation using no_terms_1
source sentence ||| translation using random_terms_1
source sentence ||| translation using terms_1

2.2. Pre-processed Submissions

For simplicity of evaluation, we split each submission into three files (according to translation mode) and provided it with the according terminology. Thus, in the split_submissions folder, the resulting .tsv files are organized as follows:

src              \t  ref               \t  terms      \t  mt
source sentence  \t reference sentence \t terminology \t MT system output

For base mode, we filled the terms column with the correct terminology (i.e., from term mode) for the sake of consistency.

3. Sumbisssion Description

3.1. General Overview

Here, we specify the relation between the sumbitted systems, their references in the paper and the necessary notes.

3.2. Languages Covered

Not all systems covered all languages (and not both datasets). In the table below, we show which system covered which data (a cross means the data are covered). Below, the columns are called the following way:

• the b: prefix means the blind dataset
• the t: prefix means the test dataset

4. For further questions

If you have any uncertainty about the submissions their preprocessing, please contact Kirill Semenov by email kir ~dot~ semenow ~at~ yandex ~dot~ ru