Batch project, 2 of 3
Log Analytics ETL
An Airflow and PySpark pipeline that turns 3.5 GB of raw web server logs into clean Parquet and reports.
Problem
Raw access logs are slow to query and expensive to keep as text. The questions people ask of them (who is hitting us, what fails, how much is bots) need structure first.
What I built
Airflow DAGs split the log by date, parse each day with PySpark, join hostnames from a 258K row lookup, apply quality rules and write daily analytics reports. New files are picked up automatically.
Result
10.36 million records processed end to end in about 5 minutes, and Parquet cut the data from 3.5 GB to 235 MB.
The numbers, and where each one comes from
Rows parsed from the access log. 10,241,992 remain after cleaning, about 1.2% removed.
from READMEREADME.md#processing-performanceSize of the raw text log against the parsed Parquet output, 93% smaller.
from READMEREADME.md#data-compressionWall clock time for all four stages on a local Docker setup.
from READMEREADME.md#processing-performanceRows in the IP to hostname lookup used by the enrichment DAG.
from codelookup/client_hostname.csvArchitecture
Raw Logs (3.5 GB)
-> split by date 5 daily files
-> parse with PySpark 235 MB Parquet, 93% smaller
-> enrich with hostname 258,445 row lookup
-> quality and cleaning 10,241,992 rows kept
-> analytics JSON reports per dayDecisions and trade-offs
Split by date first
Daily files let Spark parse days independently and let Airflow pick up a new day as soon as it lands.
Columnar storage early
Writing Parquet right after parsing means every later stage reads 93% less data.
Quality as its own stage
Duplicates, malformed IPs, invalid status codes and nulls are handled in one DAG, so analytics only reads clean data.