guillem/dr
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases 23 Wiki Activity

Add 'Copy data to postgresql'

Guillem Borrell Nogueras 2023-01-17 14:09:42 +01:00
parent dad125530a
commit e45a729db6
1 changed files with 41 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
Copy-data-to-postgresql.md Normal file

@ -0,0 +1,41 @@
```
$ head -n 100 /data/raw/data.csv | iconv -c -f ASCII -t UTF-8 | target/debug/dr csv -i | target/debug/dr schema -p -n customer_base_data
CREATE TABLE IF NOT EXISTS "customer_base_data" ( );
ALTER TABLE "customer_base_data" ADD COLUMN "SWO_Region" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "CompanyID" integer;
ALTER TABLE "customer_base_data" ADD COLUMN "CDG_Code" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "CDG_Name" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "SelltoCustomerNo" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Column1" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "ContactNo" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "CustomerName" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Customer Region" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Customer Country" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Customer ProfitCenter" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "AccountManager" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "First_Inv_Date" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Last_Inv_Date" varchar(128);
ALTER TABLE "customer_base_data" ADD COLUMN "Count_Inv" integer;
ALTER TABLE "customer_base_data" ADD COLUMN "2020" real;
ALTER TABLE "customer_base_data" ADD COLUMN "2021" real;
ALTER TABLE "customer_base_data" ADD COLUMN "2022" real;
```
We can pipe that with
```
head -n 100 /data/raw/BCG\ Report\ 01\ Customer\ Base\ Data\ -\ 2023-01-16.csv | iconv -c -f ASCII -t UTF-8 | target/debug/dr csv -i | target/debug/dr schema -p -n customer_base_data | psql -U postgres -h localhost
```
Then we want to create a new file striping the header
```
tail -n +2 /data/raw/data.csv | iconv -c -f ASCII -t UTF-8 > data.csv
```
And finally we can insert the file
```
psql -U postgres -h localhost -c "\copy customer_base_data from 'data.csv' with (FORMAT 'csv', DELIMITER ',', QUOTE '\"')"
```