Big Data Solutions in Cloud (EMR, Dataflow) Questions and Answers for Viva

Answer-1: Amazon EMR (Elastic MapReduce) is a cloud-based big data platform for processing massive amounts of data using open-source tools like Hadoop, Spark, Hive, and HBase.

Answer-2: Dataflow is a fully managed stream and batch data processing service by Google Cloud that uses Apache Beam SDK.

Answer-3: Data warehousing, big data analytics, machine learning, and log analysis.

Answer-4: ETL pipelines, real-time data processing, and event-driven architecture.

Answer-5: Hadoop is an open-source framework for distributed storage and processing of large datasets using clusters of computers.

Answer-6: Spark is an open-source distributed computing engine designed for fast computation with in-memory processing capabilities.

Answer-7: EMR supports Java, Python, Scala, and SQL (via Hive or Presto).

Answer-8: Java and Python are commonly used with the Apache Beam SDK in Dataflow.

Answer-9: EMR provisions and configures clusters automatically and allows manual or automatic scaling.

Answer-10: Hive provides a SQL-like interface to query and manage large datasets stored in EMR clusters.

Answer-11: Dataflow allows unified batch and stream processing using a single programming model (Apache Beam), unlike traditional ETL tools.

Answer-12: Yes, EMR supports auto-scaling based on metrics like CPU usage and YARN memory.

Answer-13: EMR instance groups define how different EC2 instances are assigned roles like master, core, and task nodes in a cluster.

Answer-14: The master node manages the cluster and coordinates tasks and data distribution.

Answer-15: Shuffle is the process of redistributing data across partitions, often required for joins and aggregations.

Answer-16: Batch processing handles large volumes of stored data, while stream processing analyzes data in real-time as it arrives.

Answer-17: Transforms are operations that take input data and produce output data using Apache Beam pipelines.

Answer-18: EMR uses Hadoop and Spark's native fault tolerance mechanisms like replication and lineage.

Answer-19: Dataflow handles retries, checkpointing, and dynamic work rebalancing automatically.

Answer-20: Beam is an open-source unified programming model for both batch and stream processing, used by Dataflow.

Answer-21: Amazon S3, HDFS, EMRFS, and local storage.

Answer-22: Google Cloud Storage, BigQuery, Pub/Sub, Cloud Spanner, and more.

Answer-23: EMR uses EMRFS to interact with Amazon S3 for storing input and output data.

Answer-24: BigQuery is a serverless data warehouse; Dataflow pipelines can output data directly to BigQuery for analysis.

Answer-25: A pipeline defines the series of data transformations and actions using the Apache Beam model.

Answer-26: Presto is a distributed SQL engine for querying large datasets in EMR.

Answer-27: PySpark is the Python API for Apache Spark, enabling Python developers to write Spark applications.

Answer-28: Side inputs allow additional data to be passed into a pipeline and used during processing.

Answer-29: Dynamic allocation automatically adjusts the number of executors based on workload.

Answer-30: Yes, using Spark Streaming or Apache Flink on EMR.

Answer-31: Yes, Dataflow is a serverless service that automatically provisions and manages resources.

Answer-32: The runner executes the Apache Beam pipeline. Dataflow is the runner in Google Cloud.

Answer-33: Using CloudWatch, EMR console, and Ganglia.

Answer-34: Through the Dataflow Monitoring Interface and Stackdriver (Cloud Monitoring and Logging).

Answer-35: A PCollection is the data structure in Beam that represents a collection of elements in a pipeline.

Answer-36: A web-based IDE for developing and debugging EMR applications using Jupyter notebooks.

Answer-37: Yes, Dataflow supports fixed, sliding, and session windows for grouping streaming data.

Answer-38: A DoFn (Do Function) is a Beam construct that defines how each element in a PCollection should be processed.

Answer-39: A module in Spark for structured data processing using SQL queries.

Answer-40: Yes, using Spark MLlib, TensorFlow on EMR, or integrating with SageMaker.

Answer-41: Sinks are the endpoints where processed data is written, such as BigQuery or Cloud Storage.

Answer-42: Through horizontal scaling of clusters and task distribution across nodes.

Answer-43: Dataflow automatically scales resources up or down based on pipeline needs.

Answer-44: Reusable pipeline configurations that can be parameterized and deployed without writing code.

Answer-45: A script that runs on cluster nodes when they start, used for installing software or configuring settings.

Answer-46: The time delay between data generation and result availability.

Answer-47: Yes, Dataflow can consume real-time data from Pub/Sub for stream processing.

Answer-48: EMRFS is a connector that allows Hadoop and Spark on EMR to access data stored in S3.

Answer-49: Using IAM roles, data encryption at rest/in-transit, and S3 bucket policies.

Answer-50: Through VPC Service Controls, IAM permissions, and encryption of data in transit and at rest.