Got wrong row number of dataset viewer
Hi,I’m experiencing a strange problem. I have a dataset with the typeDatasetDict, here is its basic info:DatasetDict({
    train: Dataset({
        features: ['subject', 'grade', 'skill', 'pic_choice', 'pic_prob', 'problem', 'problem_pic', 'choices', 'choices_pic', 'answer_idx'],
        num_rows: 1000
    })
    valid: Dataset({
        features: ['subject', 'grade', 'skill', 'pic_choice', 'pic_prob', 'problem', 'problem_pic', 'choices', 'choices_pic', 'answer_idx'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['subject', 'grade', 'skill', 'pic_choice', 'pic_prob', 'problem', 'problem_pic', 'choices', 'choices_pic', 'answer_idx'],
        num_rows: 1000
    })
})I use the following code to upload it into the hub:dataset_small.push_to_hub(
    hub_path, private=False, commit_message="Upload example dataset."
)After it is converted into parquet and uploaded into the hub, the dataset viewer shows the wrong number of rows:image1452×314 39.1 KBBut if I use the following code to download the dataset:from datasets import load_dataset
dataset_demo = load_dataset(hub_path)
print(dataset_demo)I will get the correct info (1k/1k/1k for each split and 3k in total).Is there something wrong with my usage ofdatasetsand huggingface hub?Update:I also got the wrong number when I used the datasets-server API inGet the number of rows and the size in bytesto get the dataset size. It seems that the backend of the datasets-server API has some bugs.Looking forward to any suggestions and thanks a lot in advance!
2024-03-13T09:49:56Z
well spotted, thanks for opening the issueThe API returns the wrong row number · Issue #2581 · huggingface/datasets-server · GitHub, we’re on it
2024-03-13T10:58:25Z
fixed. Thanks again for the investigation!
2024-03-13T12:06:22Z
Thank you for the reply and fix~
2024-06-24T03:23:05Z
Still displaying the wrong number of rows:CaptionEmporium/coyo-hd-11m-llavanext · Datasets at Hugging Face>>> len(df)
11397144It is also autogenerating parquet files for the parquets I uploaded?
2024-06-24T09:37:28Z
@lhoestq:I guess we regenerate Parquet files because of the different row group size? The number of rows is estimated on the autogenerated Parquet files, which is why it’s not exact. In this case, we could get the exact value very easily, maybe something to fix?
2024-06-24T20:53:36Z
I used thedatasetslibrary, uploaded it todata, and itstilldoesn’t give the right number of rows.import gc
import orjsonl
foo = orjsonl.load('train.jsonl.gz')
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
df = pd.DataFrame(foo)
table = pa.Table.from_pandas(df)
pq.write_table(table, './train_full.parquet')
del foo, df, table
gc.collect()
from datasets import Dataset
dataset = Dataset.from_parquet('./train_full.parquet')
dataset.save_to_disk('./huggingface')
2024-06-25T12:29:18Z
The repository currently contains three kinds of data: parquet files, json.gz, and arrow files. Moreover, the README YAML does not contain configuration to help us determine which data files should be used for which split, so: it’s best guess using heuristics.Capture d’écran 2024-06-25 à 14.26.091096×784 48.1 KBCapture d’écran 2024-06-25 à 14.26.19986×914 67.6 KBTo get the expected results, you can add aconfigs:field in the README frontmatter. We have detailed documentation pages atData files Configuration, and example datasets with the supported structures/configurations that you can replicate atdatasets-examples (Datasets examples).I hope it helps.
2024-06-25T13:15:18Z
I hope it helps.I believe so. Just for people Googling in the future.For a dataset repository to use the right data, you must consider:You must have a configuration that is YAML instead a README markdown document that points to your training splits. TheREADME.mddocument is not prepared bysave_to_disk.save_to_diskwill create adataset_info.jsonandstate.json, but that doesn’t do anything as far as the UI is concerned.The UI will ignore the file extension/files (.arrow) that are produced bysave_to_diskand instead relies on a hierarchy of extensions to find while crawling the repository.Do I have this correct? This was unexpected for myself, but if this is the way it works this is the way it works.I have updated the README.md to reflect the arrow files, but it still reports the wrong number of rows:https://huggingface.co/datasets/CaptionEmporium/coyo-hd-11m-llavanext/raw/main/README.mdhuggingface.coCaptionEmporium/coyo-hd-11m-llavanext · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-06-25T18:48:42Z
I deleted the data folder and re-uploaded only the arrow files, and only the arrow files listed in the config. That seems to have broken the auto-parquet bot.Job manager crashed while running this job (missing heartbeats).

Error code:   JobManagerCrashedErrorAt this point I’m not sure what it going wrong, but my hope without delving deeply is that at least loading the dataset using the HF library works.
2024-06-26T08:52:09Z
Note that.push_to_hub(Main classes) does all the job: upload the Dataset object to a repository on the Hub, in Parquet files, and it prepares the README.md file accordingly.For dataset-info.json and state.json, and more generally on the recommended way to prepare your dataset, I’ll let@lhoestqor@albertvillanovaanswer, as I’m not an expert.
2024-06-26T13:11:09Z
dataset-info.json and state.json are not meant to be uploaded to HFThey are part of thesave_to_disk()output which is meant for local Arrow serialization of a dataset to be reused in the same environment.You should usepush_to_hub()if you want to upload on HF