Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam braindumps download (Databricks Certified Associate Developer for Apache Spark 3.0 exam dumps Free)

QUESTION: 76

Which of the following statements about data skew is incorrect?

Spark will not automatically optimize skew joins by default.
Broadcast joins are a viable way to increase join performance for skewed data over sort-merge joins.
In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.
To mitigate skew, Spark automatically disregards null values in keys when joining.
Salting can resolve data skew.

Answer(s): D

Explanation:

To mitigate skew, Spark automatically disregards null values in keys when joining.
This statement is incorrect, and thus the correct answer to the question. Joining keys that contain null values is of particular concern with regard to data skew.
In real-world applications, a table may contain a great number of records that do not have a value assigned to the column used as a join key. During the join, the data is at risk of being heavily skewed. This is because all records with a null-value join key are then evaluated as a single large partition, standing in stark contrast to the potentially diverse key values (and therefore small partitions) of the non-null-key records.
Spark specifically does not handle this automatically. However, there are several strategies to mitigate this problem like discarding null values temporarily, only to merge them back later (see last link below).
In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.
This statement is correct. In fact, having very different partition sizes is the very definition of skew. Skew can degrade Spark performance because the largest partition occupies a single executor for a long time. This blocks a Spark job and is an inefficient use of resources, since other executors that processed smaller partitions need to idle until the large partition is processed.
Salting can resolve data skew.
This statement is correct. The purpose of salting is to provide Spark with an opportunity to repartition data into partitions of similar size, based on a salted partitioning key.
A salted partitioning key typically is a column that consists of uniformly distributed random numbers. The number of unique entries in the partitioning key column should match the number of your desired number of partitions. After repartitioning by the salted key, all partitions should have roughly the same size.
Spark does not automatically optimize skew joins by default.
This statement is correct. Automatic skew join optimization is a feature of Adaptive Query Execution (AQE). By default, AQE is disabled in Spark. To enable it, Spark's spark.sql.adaptive.enabled configuration option needs to be set to true instead of leaving it at the default false.
To automatically optimize skew joins, Spark's spark.sql.adaptive.skewJoin.enabled options also needs to be set to true, which it is by default.

When skew join optimization is enabled, Spark recognizes skew joins and optimizes them by splitting the bigger partitions into smaller partitions which leads to performance increases.
Broadcast joins are a viable way to increase join performance for skewed data over sort-merge joins. This statement is correct. Broadcast joins can indeed help increase join performance for skewed data, under some conditions. One of the DataFrames to be joined needs to be small enough to fit into each executor's memory, along a partition from the other DataFrame. If this is the case, a broadcast join increases join performance over a sort-merge join.
The reason is that a sort-merge join with skewed data involves excessive shuffling. During shuffling, data is sent around the cluster, ultimately slowing down the Spark application. For skewed data, the amount of data, and thus the slowdown, is particularly big.
Broadcast joins, however, help reduce shuffling data. The smaller table is directly stored on all executors, eliminating a great amount of network traffic, ultimately increasing join performance relative to the sort-merge join.
It is worth noting that for optimizing skew join behavior it may make sense to manually adjust Spark's spark.sql.autoBroadcastJoinThreshold configuration property if the smaller DataFrame is bigger than the 10 MB set by default. More info:
- Performance Tuning - Spark 3.0.0 Documentation
- Data Skew and Garbage Collection to Improve Spark Performance
- Section 1.2 - Joins on Skewed Data • GitBook

Show Answer Next Question

QUESTION: 77

Which of the following describes the characteristics of accumulators?

Accumulators are used to pass around lookup tables across the cluster.
All accumulators used in a Spark application are listed in the Spark UI.
Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.
Accumulators are immutable.
If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.

Answer(s): E

Explanation:

If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.
Correct, when Spark tries to rerun a failed action that includes an accumulator, it will only update the accumulator if the action succeeded.
Accumulators are immutable.
No. Although accumulators behave like write-only variables towards the executors and can only be read by the driver, they are not immutable.
All accumulators used in a Spark application are listed in the Spark UI.
Incorrect. For scala, only named, but not unnamed, accumulators are listed in the Spark UI. For pySpark, no accumulators are listed in the Spark UI – this feature is not yet implemented.
Accumulators are used to pass around lookup tables across the cluster. Wrong – this is what broadcast variables do.
Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.
Wrong, accumulators are instantiated via the accumulator(n) method of the sparkContext, for example: counter = spark.sparkContext.accumulator(0).
More info: python - In Spark, RDDs are immutable, then how Accumulators are implemented? - Stack Overflow, apache spark - When are accumulators truly reliable? - Stack Overflow, Spark – The Definitive Guide, Chapter 14

Show Answer Next Question

QUESTION: 78

Which of the following code blocks returns a DataFrame with a single column in which all items in column attributes of DataFrame itemsDf are listed that contain the letter i?

Sample of DataFrame itemsDf:

itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(attributes_exploded.contain s("i"))
itemsDf.explode(attributes).alias("attributes_exploded").filter(col("attributes_exploded").contains( "i"))
itemsDf.select(explode("attributes")).filter("attributes_exploded".contains("i"))
itemsDf.select(explode("attributes").alias("attributes_exploded")).filter(col("attributes_exploded").c ontains("i"))
itemsDf.select(col("attributes").explode().alias("attributes_exploded")).filter(col("attributes_explode d").contains("i"))

Answer(s): D

Explanation:

Result of correct code block:

To solve this question, you need to know about explode(). This operation helps you to split up arrays into single rows. If you did not have a chance to familiarize yourself with this method yet, find more examples in the documentation (link below).
Note that explode() is a method made available through pyspark.sql.functions – it is not available as a method of a DataFrame or a Column, as written in some of the answer options.
More info: pyspark.sql.functions.explode — PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, Question: 18 ( Databricks import instructions)

Show Answer Next Question

QUESTION: 79

Which of the following code blocks returns all unique values of column storeId in DataFrame transactionsDf?

transactionsDf["storeId"].distinct()
transactionsDf.select("storeId").distinct() (Correct)
transactionsDf.filter("storeId").distinct()
transactionsDf.select(col("storeId").distinct())
transactionsDf.distinct("storeId")

Answer(s): B

Explanation:

distinct() is a method of a DataFrame. Knowing this, or recognizing this from the documentation, is the key to solving this question.
More info: pyspark.sql.DataFrame.distinct — PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2, question 19 (
Databricks import instructions)

Show Answer Next Question

QUESTION: 80

Which of the following code blocks stores a part of the data in DataFrame itemsDf on executors?

itemsDf.cache().count()
itemsDf.cache(eager=True)
cache(itemsDf)
itemsDf.cache().filter()
itemsDf.rdd.storeCopy()

Answer(s): A

Explanation:

Caching means storing a copy of a partition on an executor, so it can be accessed quicker by subsequent operations, instead of having to be recalculated. cache() is a lazily-evaluated method of the DataFrame. Since count() is an action (while filter() is not), it triggers the caching process.
More info: pyspark.sql.DataFrame.cache — PySpark 3.1.2 documentation, Learning Spark, 2nd Edition, Chapter 7
Static notebook | Dynamic notebook: See test 2, Question: 20 ( Databricks import instructions)

Show Answer Next Question

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps Certified Associate Developer for Apache Spark Questions 18–80 (page: 17)

QUESTION: 76

Explanation:

QUESTION: 77

Explanation:

QUESTION: 78

Explanation:

QUESTION: 79

Explanation:

QUESTION: 80

Explanation:

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps
Certified Associate Developer for Apache Spark Questions 18–80 (page: 17)