Using Luigi for data science - d6tflow open source library

Using Luigi for data science with d6tflow

Luigi is a great library for building data workflows. But it is mostly designed for data engineering, not data science. d6flow is an open source python library built on top of Luigi, optimized for data science workflows. You can quickly:

Load task input and output into Pandas and Dask dataframes
Save Pandas and Dask dataframes to parquet, CSV or SQL
Load/save trained sklearn and keras models
Invalidate tasks including upstream/downstream tasks during trial-and-error research
Integrate with d6tpipe to quickly hand off data from data engineer to data scientist

Learn more at:

4 Reasons Why Your Machine Learning Code is Probably Bad - why use DAGs for ML projects
Using airflow-style DAGs for data science - how d6tflow is different from airflow and luigi

Getting started:

Visit Github page - star the library
Example project - go through a machine learning workflow
Readthedocs - fully documented

Github

Questions?

To learn more about the DataBolt tools and products that help you accelerate data science, check out www.databolt.tech

To see other blog posts check out our archive at blog.databolt.tech.

For questions and feedback email us at support@databolt.tech

Forward