Data and analytics on an AWS lakehouse

The AWS assessment concentrates on what the estate costs to run today and which engines each workload genuinely needs. In our experience most of the saving comes from access patterns and data layout, not from resizing clusters.

• Storage inventoried and priced. Buckets, prefixes, storage classes and lifecycle rules listed, identifying data that is stored and paid for every month but never read.
• Catalogue coverage checked. Crawler behaviour, stale table definitions and orphaned schemas compared against the prefixes actually in active use.
• Query history analysed for cost. Athena and Redshift history examined for the heaviest queries, bytes scanned and the specific reports responsible for them.
• Sources mapped with change rate. Aurora, RDS, DynamoDB and on-premises databases recorded with volume, change rate and extraction method, since that decides between batch, change data capture and zero-ETL.
• Permissions traced across all three planes. IAM policy, bucket policy and Lake Formation grants compared, with every dataset governed in more than one place documented as a risk.
• Target architecture with a costed sequence. The first workload chosen for business value rather than technical convenience, with effort and running cost attached to each step.

We build a lake with clear zones and a catalogue that means something, then add only the engines the workloads justify. The first delivery is a single business domain running end to end in production.

• Zones and conventions set at account level. Raw, curated and consumption zones in Amazon S3 with an agreed bucket and prefix convention, lifecycle policies and encryption defaults applied as account defaults rather than per bucket.
• Ingestion with reconciliation on every load. Batch ingestion built with AWS Glue and change data capture from operational databases with AWS Database Migration Service, each load producing counts that reconcile to the source.
• Curated layer in an open table format. Iceberg tables with partitioning and file sizes chosen from observed query patterns, so bytes scanned stay proportionate to the question being asked.
• Maintenance treated as part of the design. Compaction, snapshot expiry and lifecycle rules configured from the start, because small-file accumulation is what turns a cheap query into an expensive one.
• Engines added on evidence. Athena for exploration, Amazon Redshift for modelled reporting under concurrency, Amazon EMR only where Spark is genuinely required rather than culturally preferred.
• Streaming chosen on skills, not fashion. Amazon Kinesis or Amazon MSK selected on existing Kafka capability, ordering and retention requirements, with the operating burden of each stated up front.
• Everything deployed from code. Infrastructure as code across separate development, test and production accounts, so an environment can be rebuilt rather than repaired under pressure.

On AWS the governance question is where permissions live, because the same table can be reached through several engines. We settle that early and keep the answer in one place. The wider cataloguing and compliance-evidence programme is covered on our data governance and compliance page.

• One permission plane, not three. Table, column and row-level permissions centralised in AWS Lake Formation, with the duplicate bucket-level grants that quietly bypass it removed.
• Roles mapped to identity groups. Access granted to IAM roles tied to identity groups, with no long-lived user keys and no standing administrative access to production data.
• Encryption enforced, not requested. Customer-managed keys at rest and encryption in transit enforced through bucket policy rather than relying on each client being configured correctly.
• Access requests recorded as a workflow. Ownership and business context published so requesting access is a recorded request against a dataset rather than a message to an engineer who happens to be free.
• Access logging routed centrally. S3, Athena and Redshift access logs sent to the account where the security team already reviews logs, not left enabled locally and unread.
• Controls mapped to your framework. The control set documented against the framework your organisation is aligning to, such as the Essential Eight or the Australian Privacy Principles, with residual gaps recorded.

AWS gives teams more choice, and choice becomes sprawl unless somebody writes down what to use when. Enablement here is mostly about that decision and about making the catalogue the first place anyone looks.

• A written guide to engine choice. Which engine suits which job, published so a new team does not stand up its own cluster by default and then own it forever.
• Analysts trained on what a query costs. Dashboard authoring plus writing Athena queries that respect partitioning, with the cost of a full table scan made explicit in the training.
• Runbooks for pipeline operations. How to rerun a failed load, how to backfill a date range without duplicating rows, and who to call when a change data capture task falls behind.
• An owner recorded per curated dataset. Each dataset carrying a named owner and a plain-language description of what it contains, held where consumers actually look for it.
• Legacy extracts retired on sign-off. The reports and extracts the new datasets replace switched off once reconciliation has been accepted by their owner.
• Access onboarding as a process. New consumers granted access through a recorded request path instead of an ad hoc grant nobody can later explain.

Lake platforms degrade through small file accumulation, unpartitioned growth and permissions granted once for a reason nobody now remembers. We stay on to keep the platform in shape.

• Jobs and streams monitored. Glue jobs, database migration tasks and streaming consumers watched with alerting on failure, consumer lag and record counts outside expected ranges.
• Table maintenance kept current. Compaction, snapshot expiry, partitioning review and statistics refreshed so query performance does not decay as volume grows.
• Cost reported where it is generated. Monthly reporting on S3 storage by class, table maintenance charges, Athena bytes scanned and Redshift compute, with recommendations wherever spend grows faster than usage.
• Grants reviewed and pruned. Lake Formation grants and IAM role usage reviewed on an agreed cycle, with access that is no longer needed removed rather than left dormant.
• Schema and service changes tested first. Source schema changes and AWS service updates assessed in a non-production account before they touch production data.
• A visible improvement backlog. A prioritised list worked through in agreed increments alongside day-to-day support, so improvement is funded rather than deferred indefinitely.