d6tjoin - Quickly Load Any Type of CSV or Excel Data

Vendors often send large datasets in multiple files. Often there are missing and misaligned columns between files that have to be manually cleaned. With DataBolt File Stack you can easily stack them together into one consistent dataset.

Features include:

• Quickly check column consistency across multiple files
• Fix added/missing columns
• Fix renamed columns
• Check Excel tabs for consistency across files
• Quickly extract data from messy Excel files into clean CSV data
• Out of core functionality to process large files
• Export to pandas, CSV, SQL, parquet

import d6tstack.combine_csv as d6tc

c = d6tc.CombinerCSV(cfg_fnames, all_strings=True) # all_strings=True makes reading faster
col_preview = c.preview_columns()

print('all columns equal?', c.is_all_equal())
print('')
print('which columns are unique?', col_preview['columns_unique'])
print('')
print('which files have unique columns?')
print('')
print(c.is_col_present_unique())

all columns equal? False

which columns are unique? ['profit2']

which files have unique columns?

                                        profit2
filename                                       
test-data-input-csv-colmismatch-mar.csv    True
test-data-input-csv-colmismatch-feb.csv   False
test-data-input-csv-colmismatch-jan.csv   False

# quickly read all data

c.preview_combine() # keep all columns

c.preview_combine(is_col_common=True) # keep only common columns