When to Actually Store Everything in S3 and When Not To

“Just store everything in S3” has become close to a default architectural instinct in data engineering circles. It’s not bad advice. It’s also not the whole answer, and the part that gets left out is exactly where teams get burned.

The instinct comes from somewhere real: the separation of storage and compute that made modern data platforms possible in the first place. Databricks, Snowflake, and the broader Iceberg and Delta Lake ecosystem are all built on the idea that your data lives in cheap, durable object storage, and compute engines read and write to it as needed rather than owning a proprietary storage format. That’s a genuinely good default for analytical workloads, and it’s why the advice keeps circulating.

Where This Holds Up Completely

For columnar, analytical workloads, the case for object storage is close to settled. A data lake built on S3-compatible storage gives you a durable, redundant, infinitely scalable backend without managing a single disk, with strong integration across query engines, and true separation between storage and the compute that processes it. Teams running data lakes with many petabytes entirely on object storage aren’t the exception anymore, they’re increasingly the norm for this workload class.

Where It Genuinely Hits a Wall

The honest caveat, and the one that matters most, is latency, specifically for row-oriented, transactional workloads rather than analytical ones. Object storage wasn’t built for the access pattern a live application database needs: fast, small, frequent reads and writes. Once your working set is bigger than what fits in local NVMe or memory, latency against object storage directly can spike into the hundreds of milliseconds, which is a real problem for anything answering user-facing requests in real time.

This isn’t a reason to abandon object storage for transactional data. It’s a reason to be precise about what’s actually touching it directly.

How Serious Platforms Actually Solve This

The platforms that have made S3-backed storage work for low-latency, transactional workloads didn’t do it by reading and writing directly against object storage on every request. They built a caching and optimization layer in front of it. The pattern, broadly: keep the durable source of truth in object storage, but serve live reads and writes from a fast local tier, promoting and demoting data between the two as access patterns change. The application gets the low latency it needs; the storage layer still gets the durability, portability, and cost profile of object storage underneath.

That’s the actual takeaway from the “store everything in S3” advice: it’s usually shorthand for “make object storage your durable source of truth,” not “make every read and write path go directly through it.”

Mapping This to a Ceph Cluster

Ceph gives you both sides of this pattern natively, on the same cluster, rather than forcing a choice between two different platforms. Ceph’s RADOS Gateway (RGW) exposes an S3-compatible object storage API, the direct equivalent of the “store everything in object storage” pattern, well suited to data lakes, backups, media archives, and any workload where durability and portability matter more than sub-millisecond access. Ceph Block Device (RBD) exposes the same underlying cluster as low-latency block storage, the equivalent of the fast local tier a transactional workload actually needs.

On OpenMetal’s hybrid NVMe and HDD storage architecture, this isn’t a theoretical option. NVMe-backed pools handle the latency-sensitive side, HDD-backed pools handle the bulk, durable side, and both live in the same Ceph cluster under the same administrative control. You can genuinely build the pattern serious platforms use, object storage as your durable backend with a fast tier in front of it, rather than picking one interface and living with its tradeoffs everywhere.

A Simple Way to Decide

• Use object storage (RGW/S3-compatible) directly for data lakes, backups, media and file archives, and anything where access is infrequent, bulk, or already optimized by the tool reading it (most modern query engines handle S3 access patterns well)
• Use block storage (RBD) or a database for anything answering live, user-facing requests where latency matters on every single read or write
• Build a caching layer in front of object storage when you genuinely need both: a durable, portable source of truth and fast access to a working subset of it, which is exactly what a hybrid NVMe/HDD Ceph cluster is built to support
• Don’t assume the advice is wrong just because it doesn’t fit your transactional workload; it’s usually right for the analytical half of your stack even when it’s wrong for the operational half

Getting Started

For the technical detail on how OpenMetal’s hybrid storage architecture supports both patterns on one cluster, see our breakdown of storage tiers from hot to cold and our enterprise storage tier architecture guide. Current Ceph storage cluster configurations are on our storage cluster pricing page.

FAQ

Is it a good idea to store everything in S3-compatible object storage?

For analytical and data lake workloads, generally yes, it’s become close to a standard default. For transactional, latency-sensitive workloads, direct object storage access typically isn’t fast enough on its own; those workloads need a fast tier in front of the object storage layer, not a replacement for it.

Why does object storage have high latency for some workloads?

Object storage is built for durability and scale, not for the fast, frequent, small reads and writes a live application database needs. Once a workload’s active data exceeds what fits in local NVMe or memory, latency against object storage directly can climb into the hundreds of milliseconds.

What’s the difference between Ceph’s RGW and RBD interfaces?

RADOS Gateway (RGW) provides an S3-compatible object storage API, suited to data lakes, archives, and bulk or infrequent access patterns. RADOS Block Device (RBD) provides low-latency block storage from the same underlying cluster, suited to databases and other transactional workloads. Both can run on the same Ceph cluster.

Can a single storage cluster support both object storage and low-latency block storage?

Yes. A hybrid Ceph cluster with both NVMe and HDD pools can expose both RGW object storage and RBD block storage from the same underlying infrastructure, letting a single cluster serve both durable, bulk storage needs and latency-sensitive workloads.