Streaming Data in the Cloud (Kinesis, Pub/Sub) Questions and Answers for Viva

Answer-1: Streaming data is data that is continuously generated by different sources and delivered in real-time for processing.

Answer-2: AWS Kinesis is a platform for real-time data streaming that enables you to ingest, process, and analyze streaming data.

Answer-3: Google Pub/Sub is a messaging service for real-time event ingestion and delivery.

Answer-4: Real-time analytics, log monitoring, fraud detection, sensor data processing, and recommendation engines.

Answer-5: It ingests data from producers and distributes it to consumers (applications) for processing.

Answer-6: It uses a publish-subscribe model where publishers send messages to topics and subscribers receive them asynchronously.

Answer-7: Kinesis is more tightly integrated with the AWS ecosystem; Pub/Sub is native to Google Cloud and optimized for large-scale messaging.

Answer-8: Shards are units of capacity in Kinesis data streams that determine how much data you can ingest and process.

Answer-9: Topics are named resources to which messages are sent by publishers in Google Pub/Sub.

Answer-10: A Kinesis consumer is an application that reads and processes records from a data stream.

Answer-11: A Pub/Sub subscriber is a client that receives messages from a topic.

Answer-12: Yes, AWS Lambda can process events from Kinesis streams automatically.

Answer-13: Yes, Google Cloud Functions can be triggered by messages published to a Pub/Sub topic.

Answer-14: By default, Kinesis retains records for 24 hours, extendable up to 7 days.

Answer-15: Messages are retained for 7 days by default if not acknowledged.

Answer-16: Fan-out refers to sending one message to multiple subscribers of a topic.

Answer-17: Kinesis buffers data in shards and throttles based on provisioned throughput.

Answer-18: Pub/Sub automatically retries delivery until the message is acknowledged or the retention period expires.

Answer-19: A fully managed service that delivers real-time streaming data to destinations like S3, Redshift, or Elasticsearch.

Answer-20: A service to analyze streaming data using standard SQL in real-time.

Answer-21: A low-cost version of Google Pub/Sub designed for high-throughput streaming.

Answer-22: Pub/Sub supports ordering by using ordering keys so that related messages are delivered in order.

Answer-23: Yes, within a single shard, Kinesis guarantees record ordering.

Answer-24: Yes, Pub/Sub supports message filtering based on attributes.

Answer-25: No, Kinesis itself does not support filtering; you implement it in consumers.

Answer-26: A component that sends (publishes) data to a streaming service like Kinesis or Pub/Sub.

Answer-27: A component or application that reads and processes streaming data.

Answer-28: The units of data stored and transferred through a Kinesis stream.

Answer-29: The process of grouping multiple messages into a single delivery to increase efficiency.

Answer-30: Yes, producers can batch multiple records before sending them to the stream.

Answer-31: By adding more shards to a data stream.

Answer-32: It scales automatically based on message throughput and topic load.

Answer-33: Topics that store messages that could not be delivered after a maximum number of retries.

Answer-34: Using encryption, IAM roles, VPC configurations, and access policies.

Answer-35: Yes, data can be encrypted using AWS KMS.

Answer-36: Yes, it encrypts messages at rest and in transit with Google-managed or customer-managed keys.

Answer-37: Via Amazon CloudWatch metrics like incoming records, iterator age, and throughput.

Answer-38: Via Google Cloud Monitoring, showing message rates, acknowledgment latency, and errors.

Answer-39: The process of confirming that a message has been received and processed successfully.

Answer-40: Subscribers must explicitly acknowledge each message to prevent re-delivery.

Answer-41: No, Kinesis does not use acknowledgment, but consumers track their own read positions.

Answer-42: AWS SDKs for Java, Python, Node.js, and others support Kinesis.

Answer-43: Client libraries for Java, Python, Go, Node.js, and more.

Answer-44: Yes, Apache Beam can be used with Kinesis and Pub/Sub via Dataflow.

Answer-45: A standalone Java application for continuously collecting and sending data to Kinesis streams.

Answer-46: The occurrence of the same message being delivered more than once.

Answer-47: Yes, with proper configuration including enabling message ordering and acknowledgment.

Answer-48: Not directly; you need to design idempotent consumers for that.

Answer-49: Handling message duplication, ordering, latency, and scaling efficiently.

Answer-50: Managed scalability, integration with cloud services, real-time insights, and reduced infrastructure complexity.