Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam braindumps download (Databricks Certified Associate Developer for Apache Spark 3.0 exam dumps Free)

QUESTION: 71

Which of the following describes Spark's way of managing memory?

Spark uses a subset of the reserved system memory.
Storage memory is used for caching partitions derived from DataFrames.
As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
Spark's memory usage can be divided into three categories: Execution, transaction, and storage.

Answer(s): B

Explanation:

Spark's memory usage can be divided into three categories: Execution, transaction, and storage. No, it is either execution or storage.
As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
No, Spark's garbage collection runs faster on fewer big objects than many small objects. Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
The opposite is true – serialization reduces the memory footprint, but may impact performance in a negative way.
Spark uses a subset of the reserved system memory.
No, the reserved system memory is separate from Spark memory. Reserved memory stores Spark's internal objects.
More info: Tuning - Spark 3.1.2 Documentation, Spark Memory Management | Distributed Systems Architecture, Learning Spark, 2nd Edition, Chapter 7

Show Answer Next Question

QUESTION: 72

Which of the following statements about Spark's DataFrames is incorrect?

Spark's DataFrames are immutable.
Spark's DataFrames are equal to Python's DataFrames.
Data in DataFrames is organized into named columns.
RDDs are at the core of DataFrames.
The data in DataFrames may be split into multiple chunks.

Answer(s): B

Explanation:

Spark's DataFrames are equal to Python's or R's DataFrames.
No, they are not equal. They are only similar. A major difference between Spark and Python is that Spark's DataFrames are distributed, whereby Python's are not.

Show Answer Next Question

QUESTION: 73

Which of the following statements about Spark's configuration properties is incorrect?

The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
The default value for spark.sql.autoBroadcastJoinThreshold is 10MB.
The default number of partitions to use when shuffling data for joins or aggregations is 300.
The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.

Answer(s): D

Explanation:

The default number of partitions to use when shuffling data for joins or aggregations is 300. No, the default value of the applicable property spark.sql.shuffle.partitions is 200.
The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
Correct, see below.
The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
Correct, the maximum number of tasks that an executor can process in parallel depends on both properties spark.task.cpus and spark.executor.cores. This is because the available number of slots

is calculated by dividing the number of cores per executor by the number of cores per task. For more info specifically to this point, check out Spark Architecture | Distributed Systems Architecture.
More info: Configuration - Spark 3.1.2 Documentation

Show Answer Next Question

QUESTION: 74

Which of the following describes a way for resizing a DataFrame from 16 to 8 partitions in the most efficient way?

Use operation DataFrame.repartition(8) to shuffle the DataFrame and reduce the number of partitions.
Use operation DataFrame.coalesce(8) to fully shuffle the DataFrame and reduce the number of partitions.
Use a narrow transformation to reduce the number of partitions.
Use a wide transformation to reduce the number of partitions.
Use operation DataFrame.coalesce(0.5) to halve the number of partitions in the DataFrame.

Answer(s): C

Explanation:

Use a narrow transformation to reduce the number of partitions.
Correct! DataFrame.coalesce(n) is a narrow transformation, and in fact the most efficient way to resize the DataFrame of all options listed. One would run DataFrame.coalesce(8) to resize the DataFrame.
Use operation DataFrame.coalesce(8) to fully shuffle the DataFrame and reduce the number of partitions.
Wrong. The coalesce operation avoids a full shuffle, but will shuffle data if needed. This answer is incorrect because it says "fully shuffle" – this is something the coalesce operation will not do. As a general rule, it will reduce the number of partitions with the very least movement of data possible. More info: distributed computing - Spark - repartition() vs coalesce() - Stack Overflow
Use operation DataFrame.coalesce(0.5) to halve the number of partitions in the DataFrame. Incorrect, since the num_partitions parameter needs to be an integer number defining the exact number of partitions desired after the operation. More info: pyspark.sql.DataFrame.coalesce —

PySpark 3.1.2 documentation
Use operation DataFrame.repartition(8) to shuffle the DataFrame and reduce the number of partitions.
No. The repartition operation will fully shuffle the DataFrame. This is not the most efficient way of reducing the number of partitions of all listed options.
Use a wide transformation to reduce the number of partitions.
No. While possible via the DataFrame.repartition(n) command, the resulting full shuffle is not the most efficient way of reducing the number of partitions.

Show Answer Next Question

QUESTION: 75

Which of the following DataFrame operators is never classified as a wide transformation?

DataFrame.sort()
DataFrame.aggregate()
DataFrame.repartition()
DataFrame.select()
DataFrame.join()

Answer(s): D

Explanation:

As a general rule: After having gone through the practice tests you probably have a good feeling for what classifies as a wide and what classifies as a narrow transformation. If you are unsure, feel free to play around in Spark and display the explanation of the Spark execution plan via DataFrame.[operation, for example sort()].explain(). If repartitioning is involved, it would count as a wide transformation. DataFrame.select()
Correct! A wide transformation includes a shuffle, meaning that an input partition maps to one or more output partitions. This is expensive and causes traffic across the cluster. With the select() operation however, you pass commands to Spark that tell Spark to perform an operation on a specific slice of any partition. For this, Spark does not need to exchange data across partitions, each partition can be worked on independently. Thus, you do not cause a wide transformation. DataFrame.repartition()
Incorrect. When you repartition a DataFrame, you redefine partition boundaries. Data will flow across your cluster and end up in different partitions after the repartitioning is completed. This is known as a shuffle and, in turn, is classified as a wide transformation. DataFrame.aggregate()
No. When you aggregate, you may compare and summarize data across partitions. In the process, data are exchanged across the cluster, and newly formed output partitions depend on one or more input partitions. This is a typical characteristic of a shuffle, meaning that the aggregate operation may classify as a wide transformation.
DataFrame.join()
Wrong. Joining multiple DataFrames usually means that large amounts of data are exchanged across the cluster, as new partitions are formed. This is a shuffle and therefore DataFrame.join() counts as a wide transformation. DataFrame.sort()
False. When sorting, Spark needs to compare many rows across all partitions to each other. This is an expensive operation, since data is exchanged across the cluster and new partitions are formed as data is reordered. This process classifies as a shuffle and, as a result, DataFrame.sort() counts as wide transformation.
More info: Understanding Apache Spark Shuffle | Philipp Brunenberg

Show Answer Next Question

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps Certified Associate Developer for Apache Spark Questions 71–75 (page: 16)

QUESTION: 71

Explanation:

QUESTION: 72

Explanation:

QUESTION: 73

Explanation:

QUESTION: 74

Explanation:

QUESTION: 75

Explanation:

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps
Certified Associate Developer for Apache Spark Questions 71–75 (page: 16)