Dataset Card for "ted_multi"

Dataset Summary

Massively multilingual (60 language) data set derived from TED Talk transcripts. Each record consists of parallel arrays of language and text. Missing and incomplete translations will be filtered out.

Loading the dataset

The dataset is stored directly in this repository as Parquet files. No remote source archive, loading script, or trust_remote_code setting is required.

from datasets import load_dataset

dataset = load_dataset("neulab/ted_multi")
# Equivalent explicit configuration:
dataset = load_dataset("neulab/ted_multi", "plain_text")

For streaming, pass streaming=True to load_dataset.

The Parquet files were preserved without modification from conversion commit de61be957e71348e29076a9e76ed4f3b02c87e43. They contain the processed translations produced by the original loader, with missing and incomplete translations already filtered out. The original loader remains available on the script branch and in the repository history.

Supported Tasks and Leaderboards

More Information Needed

Languages

More Information Needed

Dataset Structure

Data Instances

plain_text

• Size of downloaded dataset files: 526.69 MB
• Size of the generated dataset: 791.01 MB
• Total amount of disk used: 1.32 GB

An example of 'validation' looks as follows.

This example was too long and was cropped:

{
    "talk_name": "shabana_basij_rasikh_dare_to_educate_afghan_girls",
    "translations": "{\"language\": [\"ar\", \"az\", \"bg\", \"bn\", \"cs\", \"da\", \"de\", \"el\", \"en\", \"es\", \"fa\", \"fr\", \"he\", \"hi\", \"hr\", \"hu\", \"hy\", \"id\", \"it\", ..."
}

Data Fields

The data fields are the same among all splits.

plain_text

• translations: parallel language and translation lists, with possible languages including ar, az, be, bg, bn.
• talk_name: a string feature.

Data Splits

name	train	validation	test
plain_text	258098	6049	7213

Dataset Creation

Curation Rationale

More Information Needed

Source Data

Initial Data Collection and Normalization

More Information Needed

Who are the source language producers?

More Information Needed

Annotations

Annotation process

More Information Needed

Who are the annotators?

More Information Needed

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

More Information Needed

Additional Information

Dataset Curators

More Information Needed

Licensing Information

More Information Needed

Citation Information

@InProceedings{qi-EtAl:2018:N18-2,
  author    = {Qi, Ye  and  Sachan, Devendra  and  Felix, Matthieu  and  Padmanabhan, Sarguna  and  Neubig, Graham},
  title     = {When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  month     = {June},
  year      = {2018},
  address   = {New Orleans, Louisiana},
  publisher = {Association for Computational Linguistics},
  pages     = {529--535},
  abstract  = {The performance of Neural Machine Translation (NMT) systems often suffers in low-resource scenarios where sufficiently large-scale parallel corpora cannot be obtained. Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks, which often suffer from paucity of data. However, their utility for NMT has not been extensively explored. In this work, we perform five sets of experiments that analyze when we can expect pre-trained word embeddings to help in NMT tasks. We show that such embeddings can be surprisingly effective in some cases -- providing gains of up to 20 BLEU points in the most favorable setting.},
  url       = {http://www.aclweb.org/anthology/N18-2084}
}

Contributions

Thanks to @thomwolf, @lewtun, @patrickvonplaten for adding this dataset.