Career studio / Databases
Build evidence for the Junior Data Engineer role
A training platform exports daily enrollment files. The same file may arrive twice, a column can be missing and records can be corrected later. Design a local, repeatable ingestion pipeline.
Your outcome: Local ingestion pipeline design, data-quality checks and rerun evidence.
Start with the right expectations
Who this is for
Python functions and file handling plus basic SQL; complete prerequisites before the pipeline project.
This path develops a demonstrable work sample for the named role. It does not replace employer-specific experience, professional qualifications or supervised practice.
Plan your study time
Use a flexible six-week project schedule: one week for each of the first four stages and two weeks for the portfolio and review. Allow roughly 3–5 hours per week for the case work, in addition to course study. This is a planning suggestion, not a measured course duration or a promise of job readiness.
Read the course access notes before starting. Free viewing or reading does not automatically include certificates, graded assessments, software licences or paid cloud resources. Core case work can be documented with a free editor and spreadsheet; optional implementation tools may have additional requirements.
Your course sequence
Start in the order below. Use the course outline to select the sections relevant to the stage tasks, and complete foundation exercises before moving on. Full provider courses can be substantially longer than this project schedule. These are course resources, not a claimed count of individual lessons.
Resource 01 / Harvard
CS 50 Introduction to Programming with Python
Purpose: Build the foundations. Connect your notes to the case tasks below rather than only collecting completion marks.
- Write functions, work with variables and use conditions and loops to control a program.
- Handle exceptions and debug behavior that differs from your expectations.
- Use libraries and write unit tests for reusable functions.
Access: Free CS 50 OpenCourseWare material and free development tools; free edX and GitHub accounts for assignment submission and free-certificate workflow. No required paid hosting or software.
Certificate: Free CS 50 certificate after at least 70% on each required problem and the final project. No academic credit.
Resource 02 / Harvard
CS 50’s Introduction to Databases with SQL
Purpose: Develop the role-specific method. Connect your notes to the case tasks below rather than only collecting completion marks.
- Query relational tables and connect records using primary and foreign keys.
- Model entities and relationships, choosing appropriate data types and constraints.
- Create, update and delete information while maintaining a clear database structure.
Access: Free CS 50 OpenCourseWare material and free development tools; free edX and GitHub accounts for assignment submission and free-certificate workflow. No required paid hosting or software.
Certificate: Free CS 50 certificate after at least 70% on every required problem and the final project; no automatic academic credit.
Resource 03 / IBM
Free Data Lakes Course: Cloud Lakes and Lakehouse Design
Purpose: Strengthen practical application. Connect your notes to the case tasks below rather than only collecting completion marks.
Access: Free Video
Certificate: Free video viewing; IBM certification is not included
Learn, make, check
Keep a decision log as you work. For each stage, capture the assumption, your work product, a check and the change you made after that check. The acceptance criterion is the stopping condition for the stage.
Stage 1 / suggested project week 1
Define the contract
Understand: A data contract names fields, types, keys and acceptable failures.
Make: Specify the enrollment schema, required fields and rejection rules.
Check before moving on: Document how corrected records differ from exact duplicates.
Study companion: CS 50 Introduction to Programming with Python. Use its relevant lessons and examples; the independent WikiFree task is not a provider assessment.
Save as evidence: a dated work product and a short note explaining one choice, one alternative and one remaining uncertainty.
Stage 2 / suggested project week 2
Build the local flow
Understand: Separate raw inputs from cleaned and report-ready outputs.
Make: Design or implement CSV ingestion into SQLite using staging and a stable key.
Check before moving on: Retain the raw file and an audit record for each run.
Study companion: CS 50’s Introduction to Databases with SQL. Use its relevant lessons and examples; the independent WikiFree task is not a provider assessment.
Save as evidence: a dated work product and a short note explaining one choice, one alternative and one remaining uncertainty.
Stage 3 / suggested project week 3
Make reruns safe
Understand: Idempotency means repeating a run does not duplicate its intended effect.
Make: Process the same fixture twice and record row counts and key uniqueness.
Check before moving on: A duplicate file does not create duplicate enrollments.
Study companion: Free Data Lakes Course: Cloud Lakes and Lakehouse Design. Use its relevant lessons and examples; the independent WikiFree task is not a provider assessment.
Save as evidence: a dated work product and a short note explaining one choice, one alternative and one remaining uncertainty.
Stage 4 / suggested project week 4
Handle failures visibly
Understand: A failed batch should leave a clear recovery path.
Make: Inject a missing field and invalid date; produce a quarantine report and rerun instructions.
Check before moving on: Do not silently coerce invalid records into plausible values.
Study companion: Free Data Lakes Course: Cloud Lakes and Lakehouse Design. Use its relevant lessons and examples; the independent WikiFree task is not a provider assessment.
Save as evidence: a dated work product and a short note explaining one choice, one alternative and one remaining uncertainty.
Stage 5 / suggested project weeks 5–6
Present your portfolio
Assemble the four work products into Local ingestion pipeline design, data-quality checks and rerun evidence. Write a README explaining the problem, your assumptions, the decisions you made, tests actually performed and what remains incomplete.
Give a five-minute walkthrough: one minute for the problem, two for the work and evidence, one for trade-offs and one for the next improvement. If you have not implemented a proposed system, label it a design or prototype rather than a deployed solution.
Acceptance: a reviewer can locate each work product, reproduce the key reasoning and identify the limits without a private explanation from you.
The portfolio brief
A training platform exports daily enrollment files. The same file may arrive twice, a column can be missing and records can be corrected later. Design a local, repeatable ingestion pipeline.
Deliver: Local ingestion pipeline design, data-quality checks and rerun evidence.
| enrollment id | learner id | course | updated at |
|---|---|---|---|
| E 1 | L 1 | SQL | 2026-01-02 |
| E 2 | L 2 | Python | 2026-01-02 |
| E 1 | L 1 | SQL | 2026-01-02 |
| E 3 | L 3 | SQL | invalid-date |
Use these small inputs to make the reasoning visible. They are not a representative dataset, a real client engagement or evidence of production performance. Where the brief requires additional sample records, create them yourself, label them synthetic and explain why you added them.
The ZIP includes the case CSV, four stage worksheets, course sequence, decision log, test log, portfolio README template, review rubric and reference reasoning. It is a starter kit for your work, not a completed application, professional report or proprietary project file.
Check your reasoning
The first run loads 100 unique enrollments. The identical file arrives again. What result should an idempotent load preserve?
Write your answer and assumptions before opening the reference.
Read the reference reasoning
100 unique enrollments, with the second run recorded as a rerun or no-op. The exact merge strategy depends on the documented correction policy.
Portfolio review rubric
| Criterion | Points | Evidence required |
|---|---|---|
| Problem and scope | 4 | A clear user, need, boundary and assumptions |
| Role-specific work | 8 | 2 points for each of the four stage acceptance checks |
| Reasoning and validation | 4 | Independent checks, actual results and uncertainty |
| Communication and reproducibility | 4 | A usable README, source links and reviewable work products |
Aim for at least 16/20 as a self-review target, with no missing stage acceptance check. This is a WikiFree practice rubric, not an industry credential or a hiring assessment.
Interview practice
How would you recover from a partial load without corrupting the reporting table?
Answer using the case: context, your action, evidence and limitation. Describe this honestly as a self-directed project. Do not present fictional organizations, invented interviews or practice results as paid work experience.
Your progress
Use this checklist as a personal study aid. Progress is stored in this browser when local storage is available; it is not an account record or a certificate.
0 of 5 milestones complete
The lessons and projects remain readable without JavaScript. Automatic progress saving requires JavaScript.
Questions before you start
Will this path qualify me for a job?
It helps you study relevant foundations and produce a work sample. Employers may require additional skills, qualifications and experience. Review actual role requirements before applying; no job or certification is guaranteed.
Are all tools and certificates free?
The linked learning routes were selected for free viewing or reading. Follow the separate access and certificate notes on each course. Paid tools and cloud deployment are optional; document a local design when implementation resources are unavailable.
Can I follow this on a phone?
You can read the path and many learning resources on a phone. Coding, detailed spreadsheet work, design tools and mobile-app development are easier or require a suitable computer. The page is designed to reflow; that is not a claim that every physical device has been tested.
Questions & discussion
Share a useful question or correction. Comments appear after moderation. Please avoid personal or sensitive information.