In this post we show how one can write a document ingestion RAG pipeline locally and then with minimal changes, make it scale on to systems like Ray, Dask, or PySpark.
RAG: ingestion and chunking using Hamilton…
In this post we show how one can write a document ingestion RAG pipeline locally and then with minimal changes, make it scale on to systems like Ray, Dask, or PySpark.