Free Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Braindumps
Databricks Certified Associate Developer for Apache Spark 3.5 - Python Questions 51–55 (page: 12)

Updated On: 10-Feb-2026

What is a feature of Spark Connect?

  1. It supports DataStreamReader, DataStreamWriter, StreamingQuery, and Streaming APIs
  2. Supports DataFrame, Functions, Column, SparkContext PySpark APIs
  3. It supports only PySpark applications
  4. It has built-in authentication

Answer(s): A

Explanation:

Spark Connect is a client-server architecture introduced in Apache Spark 3.4, designed to decouple the client from the Spark driver, enabling remote connectivity to Spark clusters.

According to the Spark 3.5.5 documentation:

"Majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener."

This indicates that Spark Connect supports key components of Structured Streaming, allowing for robust streaming data processing capabilities.

Regarding other options:

B .
While Spark Connect supports DataFrame, Functions, and Column APIs, it does not support SparkContext and RDD APIs.

C . Spark Connect supports multiple languages, including PySpark and Scala, not just PySpark.

D . Spark Connect does not have built-in authentication but is designed to work seamlessly with existing authentication infrastructures.



A data engineer wants to process a streaming DataFrame that receives sensor readings every second with columns sensor_id, temperature, and timestamp. The engineer needs to calculate the average temperature for each sensor over the last 5 minutes while the data is streaming.
Which code implementation achieves the requirement?

Options from the images provided:

A)



B)



C)



D)

  1. Option A
  2. Option B
  3. Option C
  4. Option D

Answer(s): D

Explanation:

The correct answer is D because it uses proper time-based window aggregation along with watermarking, which is the required pattern in Spark Structured Streaming for time-based aggregations over event-time data.

From the Spark 3.5 documentation on structured streaming:

"You can define sliding windows on event-time columns, and use groupBy along with window() to compute aggregates over those windows. To deal with late data, you use withWatermark() to specify how late data is allowed to arrive."
(Source: Structured Streaming Programming Guide)

In option D, the use of:

python

CopyEdit

.groupBy("sensor_id", window("timestamp", "5 minutes"))

.agg(avg("temperature").alias("avg_temp"))

ensures that for each sensor_id, the average temperature is calculated over 5-minute event-time windows. To complete the logic, it is assumed that withWatermark("timestamp", "5 minutes") is used earlier in the pipeline to handle late events.

Explanation of why other options are incorrect:

Option A uses Window.partitionBy which applies to static DataFrames or batch queries and is not suitable for streaming aggregations.

Option B does not apply a time window, thus does not compute the rolling average over 5 minutes.

Option C incorrectly applies withWatermark() after an aggregation and does not include any time window, thus missing the time-based grouping required.

Therefore, Option D is the only one that meets all requirements for computing a time-windowed streaming aggregation.



A data engineer is working on the DataFrame:



(Referring to the table image: it has columns Id, Name, count, and timestamp.)

Which code fragment should the engineer use to extract the unique values in the Name column into an alphabetically ordered list?

  1. df.select("Name").orderBy(df["Name"].asc())
  2. df.select("Name").distinct().orderBy(df["Name"])
  3. df.select("Name").distinct()
  4. df.select("Name").distinct().orderBy(df["Name"].desc())

Answer(s): B

Explanation:

To extract unique values from a column and sort them alphabetically:

distinct() is required to remove duplicate values.

orderBy() is needed to sort the results alphabetically (ascending by default).

Correct code:

df.select("Name").distinct().orderBy(df["Name"])

This is directly aligned with standard DataFrame API usage in PySpark, as documented in the official Databricks Spark APIs. Option A is incorrect because it may not remove duplicates. Option C omits sorting. Option D sorts in descending order, which doesn't meet the requirement for alphabetical (ascending) order.



A data scientist is working on a large dataset in Apache Spark using PySpark. The data scientist has a DataFrame df with columns user_id, product_id, and purchase_amount and needs to perform some operations on this data efficiently.

Which sequence of operations results in transformations that require a shuffle followed by transformations that do not?

  1. df.filter(df.purchase_amount > 100).groupBy("user_id").sum("purchase_amount")
  2. df.withColumn("discount", df.purchase_amount * 0.1).select("discount")
  3. df.withColumn("purchase_date", current_date()).where("total_purchase > 50")
  4. df.groupBy("user_id").agg(sum("purchase_amount").alias("total_purchase")).repartition(10)

Answer(s): D

Explanation:

Shuffling occurs in operations like groupBy, reduceByKey, or join--which cause data to be moved across partitions. The repartition() operation can also cause a shuffle, but in this context, it follows an aggregation.

In Option D, the groupBy followed by agg results in a shuffle due to grouping across nodes.

The repartition(10) is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence -- shuffle (groupBy) followed by non-shuffling (repartition) -- is correct.

Option A does the opposite: the filter does not cause a shuffle, but groupBy does -- this makes it the wrong order.



An engineer notices a significant increase in the job execution time during the execution of a Spark job. After some investigation, the engineer decides to check the logs produced by the Executors.

How should the engineer retrieve the Executor logs to diagnose performance issues in the Spark application?

  1. Locate the executor logs on the Spark master node, typically under the /tmp directory.
  2. Use the command spark-submit with the --verbose flag to print the logs to the console.
  3. Use the Spark UI to select the stage and view the executor logs directly from the stages tab.
  4. Fetch the logs by running a Spark job with the spark-sql CLI tool.

Answer(s): C

Explanation:

The Spark UI is the standard and most effective way to inspect executor logs, task time, input size, and shuffles.

From the Databricks documentation:
"You can monitor job execution via the Spark Web UI. It includes detailed logs and metrics, including task-level execution time, shuffle reads/writes, and executor memory usage." (Source: Databricks Spark Monitoring Guide)

Option A is incorrect: logs are not guaranteed to be in /tmp, especially in cloud environments.

B . --verbose helps during job submission but doesn't give detailed executor logs.

D . spark-sql is a CLI tool for running queries, not for inspecting logs.

Hence, the correct method is using the Spark UI Stages tab Executor logs.



Viewing page 12 of 28
Viewing questions 56 - 60 out of 135 questions


Post your Comments and Discuss Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 exam prep with other Community members:

AI Tutor AI Tutor 👋 I’m here to help!