CIRAL Passage Ranking Leaderboard

CIRAL is an information retrieval task with queries formulated as natural language questions in English, and retrieval at the passage level in African languages. The African languages include Hausa, Somali, Swahili and Yoruba.

We present the leaderboard of various systems evaluated on the test set; Hausa comprising of 80 queries, Somali 99 queries, Swahili 85 and Yoruba 100. Evaluation is done using judgements obtained from pooling (k=20).

Considering the nature of the task, the metrics evaluated on include nDCG@20, MRR@10, MAP and Recall@100.

Hausa Rankings


Date Run Ids Team nDCG@20 MRR@10 MAP Recall@100
26/09/2023 bm25-DT-rerank-mT5 h2oloo 0.5700 0.6952 0.3610 0.5902
26/09/2023 dt.plaid HLTCOE 0.4743 0.5846 0.3088 0.5733
26/09/2023 plaid-xlmr.mlmfine.tt HLTCOE 0.4335 0.5625 0.2711 0.5256
26/09/2023 plaid-xlmr.mlmfine.tt.jholo HLTCOE 0.3601 0.4886 0.2372 0.4829
26/09/2023 plaid-xlmr.tt HLTCOE 0.3557 0.5693 0.2468 0.5083
26/09/2023 plaid-xlmr.mlmfine.et HLTCOE 0.3488 0.5524 0.2461 0.5326
26/09/2023 plaid-xlmr.et HLTCOE 0.3481 0.5153 0.2430 0.5237
26/09/2023 bm25-dt-afrimt5-rerank h2oloo 0.3152 0.5306 0.1993 0.4186
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt2000 Masakhane 0.2830 0.4288 0.1364 0.3732
26/09/2023 afro_xlmr_base_8npassages Masakhane 0.2760 0.4921 0.1566 0.4148
26/09/2023 splade.bm25-mt5-rerank h2oloo 0.2530 0.4159 0.1384 0.2256
26/09/2023 splade-mt5-rerank h2oloo 0.2530 0.4159 0.1384 0.2256
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt1000 Masakhane 0.2451 0.3908 0.1232 0.3576
26/09/2023 afriberta_base_ckpt25k Masakhane 0.2085 0.3567 0.1153 0.3342
26/09/2023 dt.bm25-rm3 HLTCOE 0.2015 0.3571 0.1260 0.3359
26/09/2023 splade-afrimt5-rerank h2oloo 0.1771 0.3687 0.0950 0.1757
26/09/2023 afriberta_base_v2_msmarco_pft_latin_mrtydi_ft Masakhane 0.1417 0.2408 0.0738 0.2187
26/09/2023 hybrid_afriberta_dpr_splade h2oloo 0.1405 0.3143 0.0706 0.2567
26/09/2023 hybrid_mdpr_msmarco_clir_splade h2oloo 0.0895 0.2172 0.0446 0.1436
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl Masakhane 0.0895 0.1744 0.0418 0.2122
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl_ft_ckpt100 Masakhane 0.0846 0.2113 0.0376 0.1264

Somali Rankings


Date Run Ids Team nDCG@20 MRR@10 MAP Recall@100
26/09/2023 dt.plaid HLTCOE 0.5118 0.7102 0.3567 0.6436
26/09/2023 bm25-DT-rerank-mT5 h2oloo 0.4935 0.6542 0.3387 0.5581
26/09/2023 plaid-xlmr.mlmfine.tt HLTCOE 0.3366 0.5414 0.2115 0.4534
26/09/2023 plaid-xlmr.mlmfine.tt.jholo HLTCOE 0.3117 0.5249 0.1892 0.4277
26/09/2023 plaid-xlmr.et HLTCOE 0.2915 0.5513 0.1881 0.4332
26/09/2023 plaid-xlmr.tt HLTCOE 0.2878 0.4783 0.1906 0.4373
26/09/2023 plaid-xlmr.mlmfine.et HLTCOE 0.2760 0.4609 0.1911 0.4260
26/09/2023 bm25-dt-afrimt5-rerank h2oloo 0.2676 0.4680 0.1718 0.3234
26/09/2023 dt.bm25-rm3 HLTCOE 0.2550 0.3978 0.1789 0.4210
26/09/2023 splade.bm25-mt5-rerank h2oloo 0.2445 0.4440 0.1402 0.2372
26/09/2023 splade-mt5-rerank h2oloo 0.2445 0.4440 0.1402 0.2372
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt2000 Masakhane 0.2209 0.3846 0.1119 0.2975
26/09/2023 afro_xlmr_base_8npassages Masakhane 0.2165 0.4004 0.1219 0.3337
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt1000 Masakhane 0.2067 0.3840 0.1087 0.3083
26/09/2023 splade-afrimt5-rerank h2oloo 0.1647 0.3691 0.0862 0.1645
26/09/2023 hybrid_afriberta_dpr_splade h2oloo 0.1644 0.3191 0.0928 0.2467
26/09/2023 afriberta_base_ckpt25k Masakhane 0.1505 0.2845 0.0789 0.2459
26/09/2023 hybrid_mdpr_msmarco_clir_splade h2oloo 0.1198 0.2730 0.0725 0.1765
26/09/2023 afriberta_base_v2_msmarco_pft_latin_mrtydi_ft Masakhane 0.1034 0.2206 0.0436 0.1663
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl Masakhane 0.0965 0.1713 0.0546 0.1780
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl_ft_ckpt100 Masakhane 0.0830 0.1589 0.0453 0.1407

Swahili Rankings


Date Run Ids Team nDCG@20 ::after MRR@10 MAP Recall@100
26/09/2023 bm25-DT-rerank-mT5 h2oloo 0.5232 0.7222 0.3110 0.5473
26/09/2023 dt.plaid HLTCOE 0.4932 0.5995 0.3117 0.5956
26/09/2023 plaid-xlmr.mlmfine.tt HLTCOE 0.4230 0.6175 0.2500 0.4645
26/09/2023 plaid-xlmr.mlmfine.tt.jholo HLTCOE 0.4081 0.6002 0.2362 0.4477
26/09/2023 plaid-xlmr.tt HLTCOE 0.3347 0.5615 0.2041 0.4510
26/09/2023 plaid-xlmr.mlmfine.et HLTCOE 0.3301 0.5820 0.2093 0.4487
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt2000 Masakhane 0.3215 0.4983 0.1378 0.3543
26/09/2023 plaid-xlmr.et HLTCOE 0.3182 0.5541 0.1977 0.4616
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt1000 Masakhane 0.3037 0.5190 0.1302 0.3585
26/09/2023 bm25-dt-afrimt5-rerank h2oloo 0.2542 0.4791 0.1351 0.2899
26/09/2023 afro_xlmr_base_8npassages_ckpt10k Masakhane 0.2447 0.4621 0.1118 0.3499
26/09/2023 splade.bm25-mt5-rerank h2oloo 0.2395 0.4764 0.1055 0.2105
26/09/2023 splade-mt5-rerank h2oloo 0.2395 0.4764 0.1055 0.2105
26/09/2023 dt.bm25-rm3 HLTCOE 0.2178 0.4172 0.1353 0.3340
26/09/2023 afriberta_base_ckpt25k Masakhane 0.1833 0.3968 0.0792 0.2486
26/09/2023 afriberta_base_v2_msmarco_pft_latin_mrtydi_ft Masakhane 0.1426 0.3143 0.0532 0.2207
26/09/2023 splade-afrimt5-rerank h2oloo 0.1378 0.3043 0.0537 0.1215
26/09/2023 hybrid_afriberta_dpr_splade h2oloo 0.1277 0.3000 0.0565 0.1930
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl Masakhane 0.1109 0.2141 0.0463 0.1897
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl_ft_ckpt100 Masakhane 0.1068 0.2046 0.0395 0.1442
26/09/2023 hybrid_mdpr_msmarco_clir_splade h2oloo 0.0909 0.2338 0.0435 0.1807

Yoruba Rankings


Date Run Ids Team nDCG@20 MRR@10 MAP Recall@100
26/09/2023 bm25-DT-rerank-mT5 h2oloo 0.5819 0.6211 0.4512 0.8057
26/09/2023 dt.plaid HLTCOE 0.4793 0.6036 0.3657 0.7240
26/09/2023 plaid-xlmr.mlmfine.tt.jholo HLTCOE 0.4297 0.5438 0.3044 0.6748
26/09/2023 plaid-xlmr.mlmfine.tt HLTCOE 0.4189 0.5434 0.2985 0.6394
26/09/2023 bm25-dt-afrimt5-rerank h2oloo 0.4103 0.5014 0.3162 0.6377
26/09/2023 splade.bm25-mt5-rerank h2oloo 0.4071 0.5822 0.2808 0.5037
26/09/2023 splade-mt5-rerank h2oloo 0.4071 0.5822 0.2808 0.5037
26/09/2023 plaid-xlmr.mlmfine.et HLTCOE 0.3804 0.5520 0.2707 0.6950
26/09/2023 dt.bm25-rm3 HLTCOE 0.3555 0.4909 0.2696 0.6273
26/09/2023 plaid-xlmr.tt HLTCOE 0.3522 0.4731 0.2505 0.5784
26/09/2023 splade-afrimt5-rerank h2oloo 0.3138 0.4996 0.2130 0.4100
26/09/2023 plaid-xlmr.et HLTCOE 0.2627 0.4214 0.1863 0.5176
26/09/2023 afriberta_base_ckpt25k Masakhane 0.2357 0.3763 0.1423 0.4369
26/09/2023 afro_xlmr_base_8npassages Masakhane 0.2296 0.4080 0.1435 0.4418
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt2000 Masakhane 0.2210 0.4009 0.1186 0.3770
26/09/2023 hybrid_mdpr_msmarco_clir_splade h2oloo 0.2015 0.3454 0.1205 0.3786
26/09/2023 afroxlmr_base_msmarco_pft_latin_mrtydi_ft_ckpt1000 Masakhane 0.1981 0.3500 0.1071 0.3947
26/09/2023 afriberta_base_v2_msmarco_pft_latin_mrtydi_ft Masakhane 0.1688 0.2999 0.0880 0.3278
26/09/2023 hybrid_afriberta_dpr_splade h2oloo 0.1642 0.2921 0.0968 0.3452
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl Masakhane 0.1634 0.2612 0.0922 0.3658
26/09/2023 afriberta_base_ckpt25k_pft_msmarco_sw_miracl_ft_ckpt100 Masakhane 0.1594 0.2712 0.0865 0.3063