When a company like Yelp generates millions of requests every second, the data that records those requests—S3 server‑access logs—can grow faster than a squirrel on a caffeine binge. These logs are a gold mine for troubleshooting, security audits, and capacity planning, but storing and querying them at scale is a nightmare if you stick to the default “write‑once, read‑many” approach. Yelp’s recent engineering post dives into how they built a pipeline that turns raw log dumps into actionable insights without breaking the bank.

Why Traditional Log Storage Falls Short

S3 server‑access logs are essentially flat text files, each line a JSON‑like record of a single request. Storing them verbatim in a bucket means you’re paying for raw storage, which isn’t cheap when the volume scales to terabytes per day. Worse, the only way to retrieve a subset of interest is to run a full scan of the bucket, a process that can take hours and consume massive I/O bandwidth. In a production environment where uptime is paramount, waiting for a few minutes to get a simple query response is not an option.

The fundamental problem is that S3 is a key‑value store optimized for object retrieval, not for ad‑hoc analytics. Without a structured index, every query must read every object that matches the prefix, turning what should be a quick lookup into a full‑bucket scan. Yelp needed a way to keep the raw logs intact for compliance, but also to expose a high‑performance, cost‑efficient query surface.

Designing a Pipeline That Scales

The solution Yelp rolled out is a multi‑stage pipeline that mirrors the classic extract‑transform‑load (ETL) pattern, but with a twist: the transform step is engineered to reduce storage footprint while maintaining the ability to re‑hydrate the raw log when needed.

Ingestion: From S3 to a Streaming Backbone

The first hurdle is to ingest the raw logs in near‑real‑time. Yelp leverages AWS Kinesis Data Firehose to stream the S3 access logs as they are generated. This allows the pipeline to capture every request without having to poll the bucket continuously. Firehose automatically batches the records, compresses them with Parquet, and writes the resulting fragments back to a dedicated “processed” S3 bucket. The key here is that the transformation happens on the fly, keeping latency low.

Compression and Partitioning: Saving Bytes, Not Time

Parquet is a columnar format that offers tremendous compression ratios for the kind of repetitive fields found in access logs: timestamps, IP addresses, and status codes. By writing the logs in Parquet, Yelp slashes the storage cost by up to 70% compared to raw text. Moreover, the data is partitioned by date and hour, turning a monolithic file into a structured directory tree. This partitioning is critical because it lets downstream services skip entire folders when a query only asks for a narrow time window.

Cataloging with Glue: Turning Files into a Data Warehouse

Once the logs are in Parquet and partitioned, the next step is to expose them to analysts and developers. Yelp uses AWS Glue to crawl the processed bucket, infer schemas, and create a catalog. This catalog feeds into Athena, which allows ad‑hoc SQL queries without the need to move data out of S3. Because Athena reads only the partitions that match a query, the cost per query is proportional to the amount of data actually scanned.

Real‑Time Analytics with Redshift Spectrum

For workloads that require joins with other datasets or more complex aggregations, Yelp pulls the Parquet files into Redshift Spectrum. This hybrid approach lets them treat the logs as if they were a traditional relational table while still enjoying the scalability of S3. The result is a single query interface that can fetch data in milliseconds, even when the underlying dataset spans hundreds of terabytes.

Cost Control Without Sacrificing Visibility

The biggest win for Yelp is the dramatic reduction in storage and query expenses. By shifting from raw text to compressed Parquet, the raw log storage cost dropped from $0.023 per gigabyte to $0.01 per gigabyte. Athena’s pay‑per‑scan model means that the average query cost is now under a few cents, compared to the several dollars it would have cost when scanning raw logs. For a company that processes millions of requests daily, those savings translate into millions of dollars saved each year.

But Yelp didn’t just cut costs; they also preserved compliance. The raw logs are still available in the original bucket, fully immutable and backed up. In the rare event that an audit requires the exact request line, the company can retrieve the original file and decompress it on demand. The pipeline does not erase history; it simply makes the data more useful.

Lessons for the Broader Tech Community

Yelp’s approach shows that the key to managing massive log volumes is to think of logs as a two‑tiered resource: a cold, immutable archive and a hot, query‑ready layer. By combining streaming ingestion, columnar compression, partitioning, and a serverless query engine, they created a system that scales horizontally while keeping costs in check.

The same principles can be applied to other log types—application logs, security events, or IoT telemetry. The trick is to choose the right format for the job. If you can afford to spend a little time upfront on transformation, you’ll save a lot of time and money later.

What’s Next for Log Analytics?

As data volumes continue to explode, the next frontier will likely involve machine‑learning‑driven anomaly detection that runs directly on the compressed log streams. Imagine a system that flags suspicious activity in real time, without ever having to materialize the raw logs into a database. That kind of intelligence will require even tighter integration between ingestion, compression, and analytics layers—exactly the kind of architecture that Yelp has already begun to build.

So if you’re still wrestling with raw S3 logs that cost more than your coffee budget, take a page from Yelp’s playbook. Compress, partition, catalog, and let serverless analytics do the heavy lifting. The result? A lean, responsive system that lets you focus on building great experiences, not chasing down data.