Add the GSM8K dataset (#4103)

* Add the `GSM8K` dataset

* Fix yaml block

* Fix remote url for WIN

* Remove unused `os` import

* Apply suggestions from code review

Co-authored-by: jon-tow <jon-tow@Jons-MacBook-Pro.local>
Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>

 Commit from 11a50bb0b1

README.md

@@ -0,0 +1,176 @@
+---
+annotations_creators:
+- crowdsourced
+language_creators:
+- crowdsourced
+languages:
+- en
+licenses:
+- mit
+multilinguality:
+- monolingual
+size_categories:
+- 1K<n<10K
+source_datasets:
+- original
+task_categories:
+- text2text-generation
+task_ids:
+- text2text-generation-other-math-word-problems
+paperswithcode_id: gsm8k
+pretty_name: Grade School Math 8K
+---
+
+# Dataset Card for GSM8K
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-instances)
+  - [Data Splits](#data-instances)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+
+## Dataset Description
+
+- **Homepage:** https://openai.com/blog/grade-school-math/
+- **Repository:** https://github.com/openai/grade-school-math
+- **Paper:** https://arxiv.org/abs/2110.14168
+- **Leaderboard:** [Needs More Information]
+- **Point of Contact:** [Needs More Information]
+
+### Dataset Summary
+
+GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning.
+
+### Supported Tasks and Leaderboards
+
+[Needs More Information]
+
+### Languages
+
+The text in the dataset is in English. The associated BCP-47 code is `en`.
+
+## Dataset Structure
+
+### Data Instances
+
+For the `main` configuration, each instance contains a string for the grade-school level math question and a string for the corresponding answer with multiple steps of reasoning and calculator annotations (explained [here](https://github.com/openai/grade-school-math#calculation-annotations)).
+
+
+```python
+{
+    'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?',
+    'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72',
+}
+```
+
+For the `socratic` configuration, each instance contains a string for a grade-school level math question, a string for the corresponding answer with multiple steps of reasoning, calculator annotations (explained [here](https://github.com/openai/grade-school-math#calculation-annotations)), and *Socratic sub-questions*.
+
+```python
+{
+    'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?',
+    'answer': 'How many clips did Natalia sell in May? ** Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nHow many clips did Natalia sell altogether in April and May? ** Natalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72',
+}
+```
+
+### Data Fields
+
+The data fields are the same among `main` and `socratic` configurations and their individual splits.
+
+- question: The question string to a grade school math problem.
+
+- answer: The full solution string to the `question`. It contains multiple steps of reasoning with calculator annotations and the final numeric solution.
+
+### Data Splits
+
+| name   |train|validation|
+|--------|----:|---------:|
+|main    | 7473|      1319|
+|socratic| 7473|      1319|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[Needs More Information]
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+From the paper:
+
+> We initially collected a starting set of a thousand problems and natural language solutions by hiring freelance contractors on Upwork (upwork.com). We then worked with Surge AI (surgehq.ai), an NLP data labeling platform, to scale up our data collection. After collecting the full dataset, we asked workers to re-solve all problems, with no workers re-solving problems they originally wrote. We checked whether their final answers agreed with the original solu- tions, and any problems that produced disagreements were either repaired or discarded. We then performed another round of agreement checks on a smaller subset of problems, finding that 1.7% of problems still produce disagreements among contractors. We estimate this to be the fraction of problems that con- tain breaking errors or ambiguities. It is possible that a larger percentage of problems contain subtle errors.
+
+#### Who are the source language producers?
+
+[Needs More Information]
+
+### Annotations
+
+#### Annotation process
+
+[Needs More Information]
+
+#### Who are the annotators?
+
+Surge AI (surgehq.ai)
+
+### Personal and Sensitive Information
+
+[Needs More Information]
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[Needs More Information]
+
+### Discussion of Biases
+
+[Needs More Information]
+
+### Other Known Limitations
+
+[Needs More Information]
+
+## Additional Information
+
+### Dataset Curators
+
+[Needs More Information]
+
+### Licensing Information
+
+The GSM8K dataset is licensed under the [MIT License](https://opensource.org/licenses/MIT).
+
+### Citation Information
+
+```bibtex
+@article{cobbe2021gsm8k,
+  title={Training Verifiers to Solve Math Word Problems},
+  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
+  journal={arXiv preprint arXiv:2110.14168},
+  year={2021}
+}
+```
+
+### Contributions
+
+Thanks to [@jon-tow](https://github.com/jon-tow) for adding this dataset.

dataset_infos.json

@@ -0,0 +1 @@
+{"main": {"description": "GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality\nlinguistically diverse grade school math word problems. The\ndataset was created to support the task of question answering\non basic mathematical problems that require multi-step reasoning.\n", "citation": "@misc{cobbe2021training,\n      title={Training Verifiers to Solve Math Word Problems},\n      author={Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman},\n      year={2021},\n      eprint={2110.14168},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n", "homepage": "https://openai.com/blog/grade-school-math", "license": "MIT", "features": {"question": {"dtype": "string", "id": null, "_type": "Value"}, "answer": {"dtype": "string", "id": null, "_type": "Value"}}, "post_processed": null, "supervised_keys": null, "task_templates": null, "builder_name": "gsm8k", "config_name": "main", "version": {"version_str": "1.1.0", "description": null, "major": 1, "minor": 1, "patch": 0}, "splits": {"train": {"name": "train", "num_bytes": 3963202, "num_examples": 7473, "dataset_name": "gsm8k"}, "test": {"name": "test", "num_bytes": 713732, "num_examples": 1319, "dataset_name": "gsm8k"}}, "download_checksums": {"https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/train.jsonl": {"num_bytes": 4166206, "checksum": "17f347dc51477c50d4efb83959dbb7c56297aba886e5544ee2aaed3024813465"}, "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl": {"num_bytes": 749738, "checksum": "3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14"}}, "download_size": 4915944, "post_processing_size": null, "dataset_size": 4676934, "size_in_bytes": 9592878}, "socratic": {"description": "GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality\nlinguistically diverse grade school math word problems. The\ndataset was created to support the task of question answering\non basic mathematical problems that require multi-step reasoning.\n", "citation": "@misc{cobbe2021training,\n      title={Training Verifiers to Solve Math Word Problems},\n      author={Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman},\n      year={2021},\n      eprint={2110.14168},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n", "homepage": "https://openai.com/blog/grade-school-math", "license": "MIT", "features": {"question": {"dtype": "string", "id": null, "_type": "Value"}, "answer": {"dtype": "string", "id": null, "_type": "Value"}}, "post_processed": null, "supervised_keys": null, "task_templates": null, "builder_name": "gsm8k", "config_name": "socratic", "version": {"version_str": "1.1.0", "description": null, "major": 1, "minor": 1, "patch": 0}, "splits": {"train": {"name": "train", "num_bytes": 5198108, "num_examples": 7473, "dataset_name": "gsm8k"}, "test": {"name": "test", "num_bytes": 936859, "num_examples": 1319, "dataset_name": "gsm8k"}}, "download_checksums": {"https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/train_socratic.jsonl": {"num_bytes": 5401739, "checksum": "153d86551187cfd64ef7afb59bfd0ef75cea3ae9388e7ad31e43920b6dd77872"}, "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test_socratic.jsonl": {"num_bytes": 972978, "checksum": "c96673362fa7a699f4836a9b6474a067448f95fe58064727501ee63ba4c3fdb6"}}, "download_size": 6374717, "post_processing_size": null, "dataset_size": 6134967, "size_in_bytes": 12509684}}

gsm8k.py

@@ -0,0 +1,135 @@
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Grade School Math 8k dataset."""
+
+import json
+import textwrap
+
+import datasets
+
+
+_CITATION = """\
+@misc{cobbe2021training,
+      title={Training Verifiers to Solve Math Word Problems},
+      author={Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman},
+      year={2021},
+      eprint={2110.14168},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+"""
+
+_DESCRIPTION = """\
+GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality
+linguistically diverse grade school math word problems. The
+dataset was created to support the task of question answering
+on basic mathematical problems that require multi-step reasoning.
+"""
+
+_HOMEPAGE = "https://openai.com/blog/grade-school-math"
+
+_LICENSE = "MIT"
+
+_BASE_URL = "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/"
+
+
+class Gsm8kConfig(datasets.BuilderConfig):
+    """BuilderConfig for GSM8K."""
+
+    def __init__(self, urls, **kwargs):
+        """BuilderConfig for GSM8K.
+
+        Args:
+        urls: *dict[string]*, the urls for each split of the GSM8k set.
+        """
+        super().__init__(version=datasets.Version("1.1.0"), **kwargs)
+        self.urls = urls
+
+
+class Gsm8k(datasets.GeneratorBasedBuilder):
+    """Grade School Math 8k (GSM8K)"""
+
+    BUILDER_CONFIGS = [
+        Gsm8kConfig(
+            name="main",
+            description=textwrap.dedent(
+                """
+                It is segmented into 7.5K training problems and 1K test problems.
+                These problems take between 2 and 8 steps to solve, and solutions
+                primarily involve performing a sequence of elementary calculations
+                using basic arithmetic operations (+ - / *) to reach the final
+                answer. A bright middle school student should be able to solve
+                every problem.
+                """,
+            ),
+            urls={
+                "train": _BASE_URL + "train.jsonl",
+                "test": _BASE_URL + "test.jsonl",
+            },
+        ),
+        Gsm8kConfig(
+            name="socratic",
+            description=textwrap.dedent(
+                """
+                Additionally, there is a modified solution format that injects
+                automatically generated "Socratic subquestions" before each step.
+                """
+            ),
+            urls={
+                "train": _BASE_URL + "train_socratic.jsonl",
+                "test": _BASE_URL + "test_socratic.jsonl",
+            },
+        ),
+    ]
+
+    def _info(self):
+        features = datasets.Features(
+            {
+                "question": datasets.Value("string"),
+                "answer": datasets.Value("string"),
+            }
+        )
+        return datasets.DatasetInfo(
+            description=_DESCRIPTION,
+            features=features,
+            homepage=_HOMEPAGE,
+            license=_LICENSE,
+            citation=_CITATION,
+        )
+
+    def _split_generators(self, dl_manager):
+        data_dir = dl_manager.download_and_extract(self.config.urls)
+        return [
+            datasets.SplitGenerator(
+                name=datasets.Split.TRAIN,
+                gen_kwargs={
+                    "filepath": data_dir["train"],
+                },
+            ),
+            datasets.SplitGenerator(
+                name=datasets.Split.TEST,
+                gen_kwargs={
+                    "filepath": data_dir["test"],
+                },
+            ),
+        ]
+
+    def _generate_examples(self, filepath):
+        with open(filepath, encoding="utf-8") as f:
+            for key, row in enumerate(f):
+                data = json.loads(row)
+                yield key, {
+                    "question": data["question"],
+                    "answer": data["answer"],
+                }