As information on the web continually expands across different languages, Cross-lingual information retrieval (CLIR) systems which enable users search in one language and retrieve documents in another are becoming increasingly important. Research in CLIR for African languages is also growing, and these methods often require African CLIR test collections to adequately evaluate systems and expand research. Collections have been curated which either include some African languages or solely focus on African languages, however, these collections are mostly created via translation or synthetically and could be prone to bias and translation issues.
The goal of the CIRAL track is promote the research and evaluation of CLIR for African languages. With the intent of curating a human-annotated test collection through a community shared task, our track entails retrieval between English and four African languages which are Hausa, Somali, Swahili and Yoruba. Given the low-resourced nature of African languages, this track also focuses on fostering CLIR research and evaluation in low-resource settings, and hence the development of retrieval systems that are well suited for such tasks.
Track participants are tasked with developing retrieval systems that return documents in a specified African language when issued a query in English. Retrieval is done at the passage level, with queries formulated as natural language questions and passages relevant to a given query are those with answers to the question. More details on the training and tests sets are provided in the Dataset section.
Each team is required to submit run files obtained from their retrieval systems in the 6 column standard TREC format. Submissions would be recieved according to the ranking by the team and participating teams are encouraged to make a minimum of 2 submissions for each of the languages. At least the top 3 ranked submissions from each team would be included in the pools. Up to 1000 passages can be retreived per query, results with more than 1000 would be truncated. Run files can be submitted using this form
Evaluation is done by creating pools for each query and manually judging for the binary relevance of retrieved passages (pooling depth is k = 20). Using the provided judgements, the submitted run files are evaluated with the following standard retrieval metrics - Recall@100 and nDCG@20.
For each language, a static collection of passages extracted from news articles is provided. The training set comprises of the static collection, approximately 10 queries per language and some binary relevance judgements for each query. The queries and qrels in the train set serve as samples to help analyze relevance, and explore approaches while using the qrels for evaluation. The statistics of the collection is documented in the dataset repo, and can also be found in the table on the right.
The datasets would be made available in respective Hugging Face repos (Corpus, Queries and Jugdements) according to the release date for each set. Participants can request for access to the training and test sets.
| # Train Queries | # Test Queries | # Passages | |
|---|---|---|---|
| Hausa (hau) | 10 | 85 | 715,355 |
| Somali (som) | 10 | 100 | 1,015,567 |
| Swahili (swa) | 10 | 85 | 981,658 |
| Yoruba (yor) | 10 | 100 | 82,095 |
| Track Website opens | |
|---|---|
| Registration for Track Begins | |
| Training Data Released | |
| Test Data Released | |
| Run Submission Deadline | |
| Declaration of Results | |
| Working Note Submission | |
| Final Version of Working Note |
| Run results are ready and are available on the leaderboard! |
|---|
| Test queries out now: Dataset Repo |
| Check out baselines here |
| Just starting out with IR/CIRAL Quick Start, Documentation |
| Train queries and qrels released for Swahili and Somali here |
| Updated version of corpora available in Hugging Face repo |
| Training data released for Hausa and Yoruba here |
| Corpora available here |
University of Waterloo
University of Waterloo
University of Waterloo
Huawei Noah's Ark Lab
University of Waterloo
Huawei Noah's Ark Lab
Huawei Noah's Ark Lab