@ -16,21 +16,36 @@ You can install dr the rust way with `cargo install dr` but downloading a binary
```
$ dr --help
Command-line data file processing in Rust
dr is a handy command line tool to handle csv and parquet files.
It is designed to integrate nicely with other command line tools
like cat, sed, awk and database clients cli. You can find more
information an a short tutorial https://git.guillemborrell.es/guillem/dr
Usage: dr [COMMAND]
Commands:
csv Read csv, output arrow stream
sql Runs a sql statement on the file
print Pretty prints the table
rpq Read parquet file
wpq Write to a paquet file
help Print this message or the help of the given subcommand(s)
csv
Read csv, output arrow stream
schema
Several table schema related utilities
sql
Runs a sql statement on the file
print
Pretty prints the table
rpq
Read parquet file
wpq
Write to a paquet file
help
Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help information
-V, --version Print version information
-h, --help
Print help information (use `-h` for a summary)
-V, --version
Print version information
```
## Howto
@ -111,6 +126,39 @@ $ dr rpq data/yellow_tripdata_2014-01.parquet \
└─────────┴─────────────────┘
```
### Operate with SQL databases
How many times did you have to insert a csv file (sometimes larger than memory) to a database? Tens of times? Hundreds? You've probably used Pandas for that, since it can infer the table's datatypes. So a simple data operation becomes a python script with Pandas and a driver for PostgreSQL as dependencies.
Now dr can provide the table creation statement with a handful of columns:
```
$ head wine.csv | dr schema -i -p -n wine
CREATE TABLE IF NOT EXISTS "wine" ( );
ALTER TABLE "wine" ADD COLUMN "Wine" integer;
ALTER TABLE "wine" ADD COLUMN "Alcohol" real;
ALTER TABLE "wine" ADD COLUMN "Malic.acid" real;
ALTER TABLE "wine" ADD COLUMN "Ash" real;
ALTER TABLE "wine" ADD COLUMN "Acl" real;
ALTER TABLE "wine" ADD COLUMN "Mg" integer;
ALTER TABLE "wine" ADD COLUMN "Phenols" real;
ALTER TABLE "wine" ADD COLUMN "Flavanoids" real;
ALTER TABLE "wine" ADD COLUMN "Nonflavanoid.phenols" real;
ALTER TABLE "wine" ADD COLUMN "Proanth" real;
ALTER TABLE "wine" ADD COLUMN "Color.int" real;
ALTER TABLE "wine" ADD COLUMN "Hue" real;
ALTER TABLE "wine" ADD COLUMN "OD" real;
ALTER TABLE "wine" ADD COLUMN "Proline" integer;
```
More about this in the Examples section
Since most databases can ingest and spit CSV files, some simple operations can be enhanced with dr, like storing the results of a query in a parquet file
```
$ psql -c "copy (select * from wine) to stdout with (FORMAT 'csv', HEADER)" | dr csv -i -P wine.pq
```
## Reference
Some commands that generate raw output in ipc format.
@ -125,7 +173,42 @@ Some commands that convert raw input in ipc format
* Read from stdin in ipc and pretty print the table: `dr print`
* Read from stdin in csv and pretty print the table: `dr print -t`
* Read from stdin in ipc and write the data in parquet: `dr wpq [file.pq]`
* Read from stdin in ipc and write the data in parquet: `dr wpq [file.pq]`
Some commands that read csv data from stdin
* Read csv from stdin and print the schema as it would be inserted in a postgresql database: `dr schema -i -p -n tablename`
* Reas csv from stdin and save as parquet, inferring types: `dr csv -i -P filename.pq`
## Examples
### Inserting CSV into postgres
Assume that you were given a large (several GiB) with a weird (latin1) encoding, and you want to insert it into postgres. This dataset may be too large to store it in memory in one go, so you'd like to stream it into the database. You need to
* Read the csv file
* Infer the schema, and create a table
* Change the encoding of the file to the same as the database
You can use `dr` to turn this into a two-step process, and pipe the encoding conversion in one go. The first step would be to infer the schema of the resulting table and creating the table
The ingestion process is atomic, meaning that if `pgsql` fails to insert any record, no insertions will be made at all. If the insertion fails, probably because some column of type varchar can't fit the inferred type, you can change the type with:
```
$ psql -U username -h hostname -c 'alter table tablename alter column "LongDescription" type varchar(1024);' database
```
And try inserting again
## Performance
@ -142,7 +225,7 @@ On a very very old machine (Intel(R) Core(TM) i5-6500T CPU @ 2.50GHz), this take
## Caveats
1. `dr` uses Polars to build and transform dataframes in Rust, and the entire table has to be loaded in memory. At the time when `dr` was created, streaming support didn't get along very well with SQL contexts.
1. `dr` uses Polars to build and transform dataframes in Rust, and the entire table may be loaded in memory. At the time when `dr` was created, streaming support didn't get along very well with SQL contexts.
2. `dr` uses Polars' SQLContext to execute the query which supports a small subset of the SQL language.