π Python Generators & MySQL Project π
Welcome to the Python Generators & MySQL Efficient Data Processing project! This project demonstrates how to:
Connect to a MySQL database from Python
Create and seed a table with user data from a CSV file
Use Python generators to efficiently stream data from the database
File/Folder
Purpose
python-generators-0x00/seed.py
Main script: DB setup, CSV import, and generator logic
python-generators-0x00/user_data.csv
Original user data (no user_id)
python-generators-0x00/user_data_with_id.csv
User data with unique user_id (generated by update.py)
python-generators-0x00/update.py
Script to add unique user_id to each CSV row
python-generators-0x00/README.md
Project documentation for the generator project
README.md
This root documentation file
Connects to MySQL (edit credentials/port in seed.py if needed)
Creates a database and a user_data table if they don't exist
Use update.py to generate user_data_with_id.csv with unique user_ids
seed.py clears the table and inserts all users from the CSV
3. Generator for Efficient Data Streaming
(To be implemented) A generator function will yield user records one at a time from the database, saving memory and enabling batch processing.
π§βπ» Example Table Structure
user_id (PK)
name
email
age
123e4567-e89b-12d3-a456-426614174000
John Doe
john@example.com
30
...
...
...
...
Generate CSV with IDs:
python python-generators-0x00/update.py
Seed the database:
python python-generators-0x00/seed.py
(Coming soon) Stream data with a generator:
Use the generator function in seed.py to process users one by one.
Make sure MySQL is running and accessible on the configured port.
Update credentials in seed.py as needed.
For large datasets, using generators is much more memory efficient!
β¨ Features & Learning Points
Python DB connection & error handling
Table creation and data seeding
Unique ID generation for each user
Efficient data streaming with generators
Clear, emoji-rich commit messages for every step