Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam braindumps download (Databricks Certified Associate Developer for Apache Spark 3.0 exam dumps Free)

QUESTION: 61

The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1. from pyspark.sql.functions import udf
2. from pyspark.sql import types as T 3.
4. transactionsDf.createOrReplaceTempView('transactions') 5.
6. def pow_5(x):
7. return x**5 8.
9. spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10. spark.sql('SELECT power_5_udf(value) FROM transactions')

The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.
The returned DataFrame includes multiple columns instead of just one column.
The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is

not result.

Answer(s): D

Explanation:

Correct code block:
from pyspark.sql.functions import udf from pyspark.sql import types as T
transactionsDf.createOrReplaceTempView('transactions') def pow_5(x):
if x:
return x**5 return x
spark.udf.register('power_5_udf', pow_5, T.LongType()) spark.sql('SELECT power_5_udf(value) AS result FROM transactions')
Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since

Python's ** operator cannot deal with any null value Spark passes into method pow_5.
The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below).
Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument.
Omitting that argument, you end up with the column name power_5_udf(value) and not result. More info: pyspark.sql.functions.udf — PySpark 3.1.1 documentation

Show Answer Next Question

QUESTION: 62

Which of the following statements about Spark's execution hierarchy is correct?

In Spark's execution hierarchy, a job may reach over multiple stage boundaries.
In Spark's execution hierarchy, manifests are one layer above jobs.
In Spark's execution hierarchy, a stage comprises multiple jobs.
In Spark's execution hierarchy, executors are the smallest unit.
In Spark's execution hierarchy, tasks are one layer above slots.

Answer(s): A

Explanation:

In Spark's execution hierarchy, a job may reach over multiple stage boundaries.
Correct. A job is a sequence of stages, and thus may reach over multiple stage boundaries. In Spark's execution hierarchy, tasks are one layer above slots.
Incorrect. Slots are not a part of the execution hierarchy. Tasks are the lowest layer. In Spark's execution hierarchy, a stage comprises multiple jobs.
No. It is the other way around – a job consists of one or multiple stages. In Spark's execution hierarchy, executors are the smallest unit.
False. Executors are not a part of the execution hierarchy. Tasks are the smallest unit! In Spark's execution hierarchy, manifests are one layer above jobs.

Wrong. Manifests are not a part of the Spark ecosystem.

Show Answer Next Question

QUESTION: 63

Which of the following describes slots?

Slots are dynamically created and destroyed in accordance with an executor's workload.
To optimize I/O performance, Spark stores data on disk in multiple slots.
A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.
A slot is always limited to a single core.
Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.

Answer(s): C

Explanation:

Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.
Wrong, executors communicate with the driver directly.
Slots are dynamically created and destroyed in accordance with an executor's workload.
No, Spark does not actively create and destroy slots in accordance with the workload. Per executor, slots are made available in accordance with how many cores per executor (property spark.executor.cores) and how many CPUs per task (property spark.task.cpus) the Spark configuration calls for.
A slot is always limited to a single core.
No, a slot can span multiple cores. If a task would require multiple cores, it would have to be executed through a slot that spans multiple cores.
In Spark documentation, "core" is often used interchangeably with "thread", although "thread" is the more accurate word. A single physical core may be able to make multiple threads available. So, it is better to say that a slot can span multiple threads.
To optimize I/O performance, Spark stores data on disk in multiple slots. No – Spark stores data on disk in multiple partitions, not slots.
More info: Spark Architecture | Distributed Systems Architecture

Show Answer Next Question

QUESTION: 64

Which of the following describes the conversion of a computational query into an execution plan in Spark?

Spark uses the catalog to resolve the optimized logical plan.
The catalog assigns specific resources to the optimized memory plan.
The executed physical plan depends on a cost optimization from a previous stage.
Depending on whether DataFrame API or SQL API are used, the physical plan may differ.
The catalog assigns specific resources to the physical plan.

Answer(s): C

Explanation:

The executed physical plan depends on a cost optimization from a previous stage.
Correct! Spark considers multiple physical plans on which it performs a cost analysis and selects the final physical plan in accordance with the lowest-cost outcome of that analysis. That final physical plan is then executed by Spark.
Spark uses the catalog to resolve the optimized logical plan.
No. Spark uses the catalog to resolve the unresolved logical plan, but not the optimized logical plan. Once the unresolved logical plan is resolved, it is then optimized using the Catalyst Optimizer.
The optimized logical plan is the input for physical planning. The catalog assigns specific resources to the physical plan.
No. The catalog stores metadata, such as a list of names of columns, data types, functions, and databases. Spark consults the catalog for resolving the references in a logical plan at the beginning of the conversion of the query into an execution plan. The result is then an optimized logical plan. Depending on whether DataFrame API or SQL API are used, the physical plan may differ.
Wrong – the physical plan is independent of which API was used. And this is one of the great strengths of Spark!
The catalog assigns specific resources to the optimized memory plan. There is no specific "memory plan" on the journey of a Spark computation.
More info: Spark’s Logical and Physical plans … When, Why, How and Beyond. | by Laurent Leturgez
| datalex | Medium

Show Answer Next Question

QUESTION: 65

Which of the following describes characteristics of the Spark driver?

The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
The Spark driver processes partitions in an optimized, distributed fashion.
In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
The Spark driver's responsibility includes scheduling queries for execution on worker nodes.

Answer(s): D

Explanation:

The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
No, the Spark driver transforms operations into DAG computations itself.
If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
No. There is always a single driver per application, but one or more executors. The Spark driver processes partitions in an optimized, distributed fashion.
No, this is what executors do.
In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
Wrong. In a non-interactive Spark application, you need to create the SparkSession object. In an interactive Spark shell, the Spark driver instantiates the object for you.

Show Answer Next Question

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps Certified Associate Developer for Apache Spark Questions 61–65 (page: 14)

QUESTION: 61

Explanation:

QUESTION: 62

Explanation:

QUESTION: 63

Explanation:

QUESTION: 64

Explanation:

QUESTION: 65

Explanation:

Free Databricks Certified Associate Developer for Apache Spark 3.0 Exam Braindumps
Certified Associate Developer for Apache Spark Questions 61–65 (page: 14)