QUESTION: 17

A data scientist has written a data cleaning notebook that utilizes the pandas library, but their colleague has suggested that they refactor their notebook to scale with big data.
Which of the following approaches can the data scientist take to spend the least amount of time refactoring their notebook to scale with big data?

They can refactor their notebook to process the data in parallel.
They can refactor their notebook to use the PySpark DataFrame API.
They can refactor their notebook to use the Scala Dataset API.
They can refactor their notebook to use Spark SQL.
They can refactor their notebook to utilize the pandas API on Spark.

Answer(s): E

Explanation:

The data scientist can refactor their notebook to utilize the pandas API on Spark (now known as pandas on Spark, formerly Koalas). This allows for the least amount of changes to the existing pandas-based code while scaling to handle big data using Spark's distributed computing capabilities. pandas on Spark provides a similar API to pandas, making the transition smoother and faster compared to completely rewriting the code to use PySpark DataFrame API, Scala Dataset API, or Spark SQL.

Reference:

Databricks documentation on pandas API on Spark (formerly Koalas).

Reveal Solution Next Question

QUESTION: 18

A data scientist has defined a Pandas UDF function predict to parallelize the inference process for a single-node model:

They have written the following incomplete code block to use predict to score each record of Spark DataFrame spark_df:

Which of the following lines of code can be used to complete the code block to successfully complete the task?

predict(*spark_df.columns)
mapInPandas(predict)
predict(Iterator(spark_df))
mapInPandas(predict(spark_df.columns))
predict(spark_df.columns)

Answer(s): B

Explanation:

To apply the Pandas UDF predict to each record of a Spark DataFrame, you use the mapInPandas method. This method allows the Pandas UDF to operate on partitions of the DataFrame as pandas DataFrames, applying the specified function (predict in this case) to each partition. The correct code completion to execute this is simply mapInPandas(predict), which specifies the UDF to use without additional arguments or incorrect function calls.

Reference:

PySpark DataFrame documentation (Using mapInPandas with UDFs).

Reveal Solution Next Question

QUESTION: 19

Which of the Spark operations can be used to randomly split a Spark DataFrame into a training DataFrame and a test DataFrame for downstream use?

TrainValidationSplit
DataFrame.where
CrossValidator
TrainValidationSplitModel
DataFrame.randomSplit

Answer(s): E

Explanation:

The correct method to randomly split a Spark DataFrame into training and test sets is by using the randomSplit method. This method allows you to specify the proportions for the split as a list of weights and returns multiple DataFrames according to those weights. This is directly intended for splitting DataFrames randomly and is the appropriate choice for preparing data for training and testing in machine learning workflows.

Reference:

Apache Spark DataFrame API documentation (DataFrame Operations: randomSplit).

Reveal Solution Next Question

QUESTION: 20

A data scientist is using Spark ML to engineer features for an exploratory machine learning project.

They decide they want to standardize their features using the following code block:

Upon code review, a colleague expressed concern with the features being standardized prior to splitting the data into a training set and a test set.

Which of the following changes can the data scientist make to address the concern?

Utilize the MinMaxScaler object to standardize the training data according to global minimum and maximum values
Utilize the MinMaxScaler object to standardize the test data according to global minimum and maximum values
Utilize a cross-validation process rather than a train-test split process to remove the need for standardizing data
Utilize the Pipeline API to standardize the training data according to the test data's summary statistics
Utilize the Pipeline API to standardize the test data according to the training data's summary statistics

Answer(s): E

Explanation:

To address the concern about standardizing features prior to splitting the data, the correct approach is to use the Pipeline API to ensure that only the training data's summary statistics are used to standardize the test data. This is achieved by fitting the StandardScaler (or any scaler) on the training data and then transforming both the training and test data using the fitted scaler. This approach prevents information leakage from the test data into the model training process and ensures that the model is evaluated fairly.

Reference:

Best Practices in Preprocessing in Spark ML (Handling Data Splits and Feature Standardization).

Reveal Solution Next Question

Free Databricks-Machine-Learning-Associate Exam Braindumps (page: 5)

QUESTION: 17

Explanation:

Reference:

QUESTION: 18

Explanation:

Reference:

QUESTION: 19

Explanation:

Reference:

QUESTION: 20

Explanation:

Reference: