csarron/4m-img-caps

see read_pyarrow.py for how to read one pyarrow file. example PyTorch dataset: from torch.utils.data import Dataset class ImageCaptionArrowDataset(Dataset): def __init__( self, dataset_file, tokenizer, ): import pyarrow as pa data = [pa.ipc.open_file(pa.memory_map(f, "rb")).read_all() for f in glob.glob(dataset_file)] self.data = pa.concat_tables(data) # do other initialization, like init image preprocessing fn, def… See the full description on the dataset page: huggingface.co/datasets/csarron/4m-img-caps.