Articles on Athena, OpenSearch, Glue, data lake construction, and other AWS data analytics and search services
Amazon OpenSearch Serverless is a fully managed service that handles search and analytics workloads with auto-scaling while eliminating cluster management. This article explains the OCU billing model, collection type selection criteria, and index design best practices from a practical perspective.
Ingest real-time data with Data Streams and automatically deliver it to S3, Redshift, and OpenSearch with Data Firehose. This article explains how to build a streaming pipeline with shard design and on-demand mode selection.
Learn how to build digital twins with AWS IoT TwinMaker. This article covers 3D scene creation, IoT data integration, Grafana dashboard integration, and industrial equipment visualization.
Learn how to perform entity resolution (record matching) on customer data using AWS Entity Resolution. This article covers ML-based matching, rule-based matching, privacy protection, and integration with Clean Rooms.
Learn about privacy-enhanced data analytics with AWS Clean Rooms. Explore multi-organization data collaboration, access control through analysis rules, and differential privacy.
Design a managed Apache Kafka cluster, learn when to choose MSK Serverless, and explore data integration patterns with MSK Connect.
Explains how to build interactive BI dashboards with Amazon QuickSight and a serverless data analytics platform with Athena integration. Covers high-speed visualization with the SPICE engine and practical methods for sharing insights across the organization.
Learn the criteria for choosing between Serverless and RA3 provisioned clusters, and how to prevent data silos using data sharing and Spectrum for data lake integration.
Learn about building, access control, and governance for data lakes using AWS Lake Formation. This article covers fine-grained column-level and row-level permission management for S3-based data lakes, along with Glue and Athena integration.
Learn how to build Spark clusters with EMR, choose between EMR Serverless, and optimize costs with spot instances.
Automatically detect schemas with crawlers and execute ETL processing with Glue jobs. This guide covers Data Catalog usage patterns and visual development with Glue Studio.
Learn how to build an IoT device data collection, preprocessing, and analytics pipeline with AWS IoT Analytics. Covers the four components - channels, pipelines, data stores, and datasets - plus QuickSight integration.
Build a log analytics platform and optimize costs with index lifecycle management. Learn about OpenSearch Dashboards and Serverless mode.
Learn how to collect, model, and visualize industrial equipment data with AWS IoT SiteWise. Covers OPC-UA gateways, asset models, and SiteWise Monitor dashboards.
Achieve fast queries with the SPICE engine and integrate BI capabilities into your own applications using the embedding API. This article introduces Q's natural language queries and Reader session-based pricing.
Establish data lake governance with column-level and row-level fine-grained access control and tag-based management. This article covers integration with Glue Data Catalog and cross-account access.
Learn how to build a domain-based data catalog and implement data discovery, sharing, and access control through subscription workflows.
Learn how to run Spark jobs and Hive queries on EMR clusters, choose between EMR Serverless, and optimize costs with managed scaling.
Learn about record matching across multiple data sources with Entity Resolution and how to design matching workflows.
Learn how to select distribution styles and sort keys, design workload management (WLM), and accelerate queries with AQUA in Amazon Redshift.
Explore data lake design patterns using S3 as the storage foundation and Lake Formation for fine-grained access control. This article also covers ETL pipelines and cost optimization.
Run joint analysis across multiple companies without sharing or copying data. Learn about aggregation rules for preventing individual identification and Cryptographic Computing for encrypted analysis.
Learn about running Spark and Hive jobs with EMR Serverless, job run design, and cost optimization strategies.
Run real-time stream processing with SQL or Java/Python applications in a fully managed Apache Flink environment. This article covers design patterns for window aggregation, pattern detection, and Kinesis/MSK integration.
Learn how to collect vehicle telemetry data with AWS IoT FleetWise, including vehicle modeling, edge-side data filtering, and efficient data transfer to the cloud.
Learn how to build a cluster-free Kafka environment with IAM authentication and simplify event streaming through integration with EventBridge Pipes.
Explore concrete use cases for sales, IT, and finance departments, along with design patterns for notifications, approvals, and multi-step workflows using Quick Flows.
Learn about data search and analytics design with Amazon OpenSearch Service, including how to build an analytics foundation with full-text search, log analysis, and dashboard visualization.
Learn streaming data processing design techniques with Amazon Kinesis, including building real-time data pipelines with Data Streams, Data Firehose, and Lambda integration.
Learn how to build an IoT device management platform using AWS IoT Core and Lambda.
Explains how to build blockchain networks with Amazon Managed Blockchain and use Amazon QLDB as a verifiable ledger database. Covers practical use cases such as supply chain management and ensuring transparency in financial transactions.
Learn how to acquire and utilize third-party data with AWS Data Exchange. This article covers building data pipelines with S3 integration and publishing data as a data provider.
Learn design patterns for real-time data pipelines combining Kinesis Data Streams and Data Firehose. This article covers shard design, buffering, Lambda transformations, and delivery to S3, Redshift, and OpenSearch.
Learn how to build a fully managed Kafka cluster with Amazon MSK (Managed Streaming for Apache Kafka) and when to choose it over Kinesis. This article covers design patterns for large-scale real-time data streaming infrastructure.
Learn practical approaches to quantum computing with Amazon Braket. This article covers quantum circuit simulators, access to real quantum hardware, implementing hybrid quantum-classical algorithms, and integration patterns with Lambda.
Learn how to build a video transcoding pipeline using AWS Elemental MediaConvert and S3. This article covers multi-format output, HDR support, and cost-efficient serverless media processing techniques.
Procure third-party data products via Marketplace and build automated delivery pipelines to S3. This article also covers how to productize and monetize your own data.