Data and analytics on AWS

The AWS assessment concentrates on what the estate costs to run today and which engines each workload genuinely needs. In our experience most of the saving comes from access patterns and data layout, not from resizing clusters.

• Inventory S3 buckets, prefixes, storage classes and lifecycle rules, and identify data that is stored and paid for but never read.
• Review catalogue coverage, stale table definitions and crawler behaviour against the buckets that are actually in active use.
• Analyse Athena and Redshift query history to find the heaviest queries, the volume of bytes scanned and the specific reports responsible.
• Map source systems including Aurora, RDS, DynamoDB and on-premises databases, recording the change rate and extraction method for each.
• Assess current permissions across IAM, Lake Formation and per-engine grants, and document every dataset governed in more than one place.
• Produce a target architecture and a costed sequence, with the first workload chosen for business value rather than technical convenience.

We build a lake with clear zones and a catalogue that means something, then add only the engines the workloads justify. The first delivery is a single business domain running end to end in production.

• Establish raw, curated and consumption zones in Amazon S3 with an agreed bucket and prefix convention, lifecycle policies and encryption defaults set at account level.
• Build batch ingestion with AWS Glue and change data capture from operational databases with AWS Database Migration Service, with reconciliation counts on every load.
• Store curated data in an open table format, with partitioning and file sizes chosen from observed query patterns so bytes scanned stay proportionate to the question asked.
• Register everything in the catalogue and expose it through Athena for exploration, Amazon Redshift for modelled reporting and Amazon EMR where Spark is genuinely required.
• Stream event data through Amazon Kinesis or Amazon MSK, choosing between them on existing Kafka skills, ordering and retention requirements rather than on preference.
• Deploy the platform through infrastructure as code across separate development, test and production accounts, so an environment can be rebuilt rather than repaired.

On AWS the governance question is where permissions live, because the same table can be reached through several engines. We settle that early and keep the answer in one place.

• Centralise table, column and row-level permissions in AWS Lake Formation and remove the duplicate bucket-level grants that quietly bypass it.
• Grant access to IAM roles mapped to identity groups, with no long-lived user keys and no standing administrative access to production data.
• Encrypt data at rest with customer-managed keys and enforce encryption in transit through bucket policy rather than relying on client configuration.
• Publish business context, ownership and subscription-based access through Amazon DataZone, so requesting access becomes a recorded workflow rather than a message to an engineer.
• Enable access logging for S3, Athena and Redshift and route it to the account where the security team already reviews logs.
• Document the control set against the framework the organisation is aligning to, such as the Essential Eight or the Australian Privacy Principles, and record the residual gaps.

AWS gives teams more choice, and choice becomes sprawl unless someone writes down what to use when. Enablement here is mostly about that decision and about making the catalogue the first place anyone looks.

• Publish a written guide to which engine suits which job, so a new team does not stand up its own cluster by default.
• Train analysts on Amazon Quick and on writing Athena queries that respect partitioning, with the cost of a full scan made explicit in the training.
• Hand over runbooks for pipeline operation, including how to rerun a failed load and how to backfill a date range without duplicating rows.
• Agree the owner of each curated dataset and record it in the catalogue with a plain-language description of what it contains.
• Retire the legacy extracts and reports the new datasets replace, once reconciliation has been signed off by the owner.
• Set up an access request workflow so new consumers are onboarded through a recorded process instead of an ad hoc grant.

Lake platforms degrade through small file accumulation, unpartitioned growth and permissions granted once for a reason nobody now remembers. We stay on to keep the platform in shape.

• Monitor Glue jobs, database migration tasks and streaming consumers, alerting on failure, consumer lag and record counts outside expected ranges.
• Compact small files, review partitioning and refresh table statistics so query performance does not decay as volume grows.
• Report monthly on S3 storage by class, Athena bytes scanned and Redshift compute, with specific recommendations wherever spend grows faster than usage.
• Review Lake Formation grants and IAM role usage on an agreed cycle and remove access that is no longer needed.
• Manage source schema changes and AWS service updates, testing downstream impact in a non-production account before applying them.
• Maintain a prioritised improvement backlog and work through it in agreed increments alongside day-to-day support.