Quiz AWS Certified Data Engineer - Associate

The error message indicates that the AWS Glue job cannot access the Amazon S3 bucket through the
VPC endpoint. This could be because the VPC’s route table does not have the necessary routes to
direct the traffic to the endpoint. To fix this, the data engineer must verify that the route table has an
entry for the Amazon S3 service prefix (com.amazonaws.region.s3) with the target as the VPC
endpoint ID. This will allow the AWS Glue job to use the VPC endpoint to access the S3 bucket
without going through the internet or a NAT gateway. For more information, see Gateway
endpoints. Reference:
Troubleshoot the AWS Glue error “VPC S3 endpoint validation failed”
Amazon VPC endpoints for Amazon S3
[AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide]

AWS Lake Formation is a service that allows you to easily set up, secure, and manage data lakes. One
of the features of Lake Formation is row-level security, which enables you to control access to specific
rows or columns of data based on the identity or role of the user. This feature is useful for scenarios
where you need to restrict access to sensitive or regulated data, such as customer data from different
countries. By registering the S3 bucket as a data lake location in Lake Formation, you can use the Lake
Formation console or APIs to define and apply row-level security policies to the data in the bucket.
You can also use Lake Formation blueprints to automate the ingestion and transformation of data
from various sources into the data lake. This solution requires the least operational effort compared
to the other options, as it does not involve creating or moving data, or managing multiple tables,
views, or roles. Reference:
AWS Lake Formation
Row-Level Security
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 4: Data Lakes and
Data Warehouses, Section 4.2: AWS Lake Formation

AWS Data Exchange is a service that makes it easy to find, subscribe to, and use third-party data in
the cloud. It provides a secure and reliable way to access and integrate data from various sources,
such as data providers, public datasets, or AWS services. Using AWS Data Exchange, you can browse
and subscribe to data products that suit your needs, and then use API calls or the AWS Management
Console to export the data to Amazon S3, where you can use it with your existing analytics platform.
This solution minimizes the effort and time required to incorporate third-party datasets, as you do
not need to set up and manage data pipelines, storage, or access controls. You also benefit from the
data quality and freshness provided by the data providers, who can update their data products as
frequently as needed12.
The other options are not optimal for the following reasons:
B . Use API calls to access and integrate third-party datasets from AWS. This option is vague and does
not specify which AWS service or feature is used to access and integrate third-party datasets. AWS
offers a variety of services and features that can help with data ingestion, processing, and analysis,
but not all of them are suitable for the given scenario. For example, AWS Glue is a serverless data
integration service that can help you discover, prepare, and combine data from various sources, but
it requires you to create and run data extraction, transformation, and loading (ETL) jobs, which can
add operational overhead3.
C . Use Amazon Kinesis Data Streams to access and integrate third-party datasets from AWS
CodeCommit repositories. This option is not feasible, as AWS CodeCommit is a source control service
that hosts secure Git-based repositories, not a data source that can be accessed by Amazon Kinesis
Data Streams. Amazon Kinesis Data Streams is a service that enables you to capture, process, and
analyze data streams in real time, such as clickstream data, application logs, or IoT telemetry. It does
not support accessing and integrating data from AWS CodeCommit repositories, which are meant for
storing and managing code, not data .
D . Use Amazon Kinesis Data Streams to access and integrate third-party datasets from Amazon
Elastic Container Registry (Amazon ECR). This option is also not feasible, as Amazon ECR is a fully
managed container registry service that stores, manages, and deploys container images, not a data
source that can be accessed by Amazon Kinesis Data Streams. Amazon Kinesis Data Streams does not
support accessing and integrating data from Amazon ECR, which is meant for storing and managing
container images, not data .
Reference:
1: AWS Data Exchange User Guide
2: AWS Data Exchange FAQs
3: AWS Glue Developer Guide
: AWS CodeCommit User Guide
: Amazon Kinesis Data Streams Developer Guide
: Amazon Elastic Container Registry User Guide
: Build a Continuous Delivery Pipeline for Your Container Images with Amazon ECR as Source

A data mesh is an architectural framework that organizes data into domains and treats data as
products that are owned and offered for consumption by different teams1. A data mesh requires a
centralized layer for data governance and access control, as well as a distributed layer for data
storage and analysis. AWS Glue can provide data catalogs and ETL operations for the data mesh, but
it cannot provide data governance and access control by itself2. Therefore, the company needs to
use another AWS service for this purpose. AWS Lake Formation is a service that allows you to create,
secure, and manage data lakes on AWS3. It integrates with AWS Glue and other AWS services to
provide centralized data governance and access control for the data mesh. Therefore, option E is
correct.
For data storage and analysis, the company can choose from different AWS services depending on
their needs and preferences. However, one of the benefits of a data mesh is that it enables data to be
stored and processed in a decoupled and scalable way1. Therefore, using serverless or managed
services that can handle large volumes and varieties of data is preferable. Amazon S3 is a highly
scalable, durable, and secure object storage service that can store any type of data. Amazon Athena
is a serverless interactive query service that can analyze data in Amazon S3 using standard SQL.
Therefore, option B is a good choice for data storage and analysis in a data mesh. Option A, C, and D
are not optimal because they either use relational databases that are not suitable for storing diverse
and unstructured data, or they require more management and provisioning than serverless
services. Reference:
1: What is a Data Mesh? - Data Mesh Architecture Explained - AWS
2: AWS Glue - Developer Guide
3: AWS Lake Formation - Features
[4] : Design a data mesh architecture using AWS Lake Formation and AWS Glue
[5] : Amazon S3 - Features
[6] : Amazon Athena - Features

Lambda layers are a way to share code and dependencies across multiple Lambda functions. By
packaging the custom Python scripts into Lambda layers, the data engineer can update the scripts in
one place and have them automatically applied to all the Lambda functions that use the layer. This
reduces the manual effort and ensures consistency across the Lambda functions. The other options
are either not feasible or not efficient. Storing a pointer to the custom Python scripts in the execution
context object or in environment variables would require the Lambda functions to download the
scripts from Amazon S3 every time they are invoked, which would increase latency and cost.
Assigning the same alias to each Lambda function would not help with updating the Python scripts,
as the alias only points to a specific version of the Lambda function code. Reference:
AWS Lambda layers
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 3: Data Ingestion
and Transformation, Section 3.4: AWS Lambda

AWS Glue workflows are a cost-effective way to orchestrate complex ETL jobs that involve multiple
crawlers, jobs, and triggers. AWS Glue workflows allow you to visually monitor the progress and
dependencies of your ETL tasks, and automatically handle errors and retries. AWS Glue workflows
also integrate with other AWS services, such as Amazon S3, Amazon Redshift, and AWS Lambda,
among others, enabling you to leverage these services for your data processing workflows. AWS Glue
workflows are serverless, meaning you only pay for the resources you use, and you don’t have to
manage any infrastructure.
AWS Step Functions, AWS Glue Studio, and Amazon MWAA are also possible options for
orchestrating ETL pipelines, but they have some drawbacks compared to AWS Glue workflows. AWS
Step Functions is a serverless function orchestrator that can handle different types of data
processing, such as real-time, batch, and stream processing. However, AWS Step Functions requires
you to write code to define your state machines, which can be complex and error-prone. AWS Step
Functions also charges you for every state transition, which can add up quickly for large-scale ETL
pipelines.
AWS Glue Studio is a graphical interface that allows you to create and run AWS Glue ETL jobs without
writing code. AWS Glue Studio simplifies the process of building, debugging, and monitoring your
ETL jobs, and provides a range of pre-built transformations and connectors. However, AWS Glue
Studio does not support workflows, meaning you cannot orchestrate multiple ETL jobs or crawlers
with dependencies and triggers. AWS Glue Studio also does not support streaming data sources or
targets, which limits its use cases for real-time data processing.
Amazon MWAA is a fully managed service that makes it easy to run open-source versions of Apache
Airflow on AWS and build workflows to run your ETL jobs and data pipelines. Amazon MWAA
provides a familiar and flexible environment for data engineers who are familiar with Apache Airflow,
and integrates with a range of AWS services such as Amazon EMR, AWS Glue, and AWS Step
Functions. However, Amazon MWAA is not serverless, meaning you have to provision and pay for the
resources you need, regardless of your usage. Amazon MWAA also requires you to write code to
define your DAGs, which can be challenging and time-consuming for complex ETL
pipelines. Reference:
AWS Glue Workflows
AWS Step Functions
AWS Glue Studio
Amazon MWAA
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

The Amazon Redshift Data API is a built-in feature that allows you to run SQL queries on Amazon
Redshift data with web services-based applications, such as AWS Lambda, Amazon SageMaker
notebooks, and AWS Cloud9. The Data API does not require a persistent connection to your
database, and it provides a secure HTTP endpoint and integration with AWS SDKs. You can use the
endpoint to run SQL statements without managing connections. The Data API also supports both
Amazon Redshift provisioned clusters and Redshift Serverless workgroups. The Data API is the best
solution for running real-time queries on the financial data from within the trading application, as it
has the least operational overhead compared to the other options.
Option A is not the best solution, as establishing WebSocket connections to Amazon Redshift would
require more configuration and maintenance than using the Data API. WebSocket connections are
also not supported by Amazon Redshift clusters or serverless workgroups.
Option C is not the best solution, as setting up JDBC connections to Amazon Redshift would also
require more configuration and maintenance than using the Data API. JDBC connections are also not
supported by Redshift Serverless workgroups.
Option D is not the best solution, as storing frequently accessed data in Amazon S3 and using
Amazon S3 Select to run the queries would introduce additional latency and complexity than using
the Data API. Amazon S3 Select is also not optimized for real-time queries, as it scans the entire
object before returning the results. Reference:
Using the Amazon Redshift Data API
Calling the Data API
Amazon Redshift Data API Reference
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

Athena workgroups are a way to isolate query execution and query history among users, teams, and
applications that share the same AWS account. By creating a workgroup for each use case, the
company can control the access and actions on the workgroup resource using resource-level IAM
permissions or identity-based IAM policies. The company can also use tags to organize and identify
the workgroups, and use them as conditions in the IAM policies to grant or deny permissions to the
workgroup. This solution meets the requirements of separating query processes and access to query
history among users, teams, and applications that are in the same AWS account. Reference:
Athena Workgroups
IAM policies for accessing workgroups
Workgroup example policies

The FLEX execution class allows you to run AWS Glue jobs on spare compute capacity instead of
dedicated hardware. This can reduce the cost of running non-urgent or non-time sensitive data
integration workloads, such as testing and one-time data loads. The FLEX execution class is available
for AWS Glue 3.0 Spark jobs. The other options are not as cost-effective as FLEX, because they either
use dedicated resources (STANDARD) or do not affect the cost at all (Spot Instance type and
GlueVersion). Reference:
Introducing AWS Glue Flex jobs: Cost savings on ETL workloads
Serverless Data Integration – AWS Glue Pricing
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide (Chapter 5, page 125)

Option A is the correct answer because it meets the requirements with the least operational
overhead. Creating an S3 event notification that has an event type of s3:ObjectCreated:* will trigger
the Lambda function whenever a new object is created in the S3 bucket. Using a filter rule to
generate notifications only when the suffix includes .csv will ensure that the Lambda function only
runs for .csv files. Setting the ARN of the Lambda function as the destination for the event
notification will directly invoke the Lambda function without any additional steps.
Option B is incorrect because it requires the user to tag the objects with .csv, which adds an extra
step and increases the operational overhead.
Option C is incorrect because it uses an event type of s3:*, which will trigger the Lambda function for
any S3 event, not just object creation. This could result in unnecessary invocations and increased
costs.
Option D is incorrect because it involves creating and subscribing to an SNS topic, which adds an
extra layer of complexity and operational overhead.
Reference:
AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 3: Data Ingestion
and Transformation, Section 3.2: S3 Event Notifications and Lambda Functions, Pages 67-69
Building Batch Data Analytics Solutions on AWS, Module 4: Data Transformation, Lesson 4.2: AWS
Lambda, Pages 4-8
AWS Documentation Overview, AWS Lambda Developer Guide, Working with AWS Lambda
Functions, Configuring Function Triggers, Using AWS Lambda with Amazon S3, Pages 1-5