Hey guys, would it be possible import over 200 million records into Supabase that is coming from a text file?

This should be possible but your text file would need to be processed somehow or have to already conform to your table columns. So CSV import is something you could do. I found this article online about how to import large CSVs into Postgres https://www.tractionondemand.com/blog/using-copy-in-postgres-for-importing-large-csvs/

I concur with what silentworks said. COPY is definitely one way to go, but getting it as CSV format which has the same columns as your table is crucial. If you can manage that, then something like this, along with some sort of lib to stream the data, can help clear the table and import the new data (but won't clear the table unless the import succeeds):

sql
BEGIN;
    TRUNCATE TABLE my_table;
    COPY public.my_table (column_1, column_2, column_3) FROM STDIN CSV HEADER;
COMMIT;

Depending on the specifics of the data (e.g. for delimiting, or handling quotes in the data), you might also need to look at adding some additional info after `HEADER`:

sql
QUOTE '\"' ESCAPE ''''

Thanks a bunch @User and @User

If you're hosting it on the Supabase platform (i.e. not self-hosting), and you're not on the pay as you go plan, you're unlikely to be able to import that much data as you'll hit disk space limits and the import will fail. I know this, because I hit my limits last year, still needed more space to import even more data, and ultimately ended up switching to self-hosting to resolve it. The most reliable option for importing huge datasets is to use postgres COPY via either a streaming library, or via a dedicated tool like pgadmin.

Ok I think its best if I self host supabase then thanks @User