From S3 to GPU in One Copy
Spiral
The talk “From S3 to GPU in One Copy” explains how SpiralDB’s Vortex columnar format enables training data to be streamed from S3 into GPUs with projection and filter pruning instead of decompressing/reprocessing everything. It reports S3-to-GPU scans at around 30 times faster than Parquet. As a result, only selected columns/segments are read and decompressed for each iteration, reducing movement and decision overhead during ML training.
Why it matters
Vortex streams compressed columnar data from S3 through host memory to the GPU with a single copy, avoiding disk staging and CPU decompression bottlenecks. It prunes data before transfer and supports GPU-side decompression and computation.