A machine learning engineer is trying to scale a machine learning pipeline by distributing its feature engineering process.Which of the following feature engineering tasks will be the least efficient to distribute?
Answer(s): D
Among the options listed, calculating the true median for imputing missing feature values is the least efficient to distribute. This is because the true median requires knowledge of the entire data distribution, which can be computationally expensive in a distributed environment. Unlike mean or mode, finding the median requires sorting the data or maintaining a full distribution, which is more intensive and often requires shuffling the data across partitions.ReferenceChallenges in parallel processing and distributed computing for data aggregation like median calculation: https://www.apache.org
Which of the following is a benefit of using vectorized pandas UDFs instead of standard PySpark UDFs?
Answer(s): B
Vectorized pandas UDFs, also known as Pandas UDFs, are a powerful feature in PySpark that allows for more efficient operations than standard UDFs. They operate by processing data in batches, utilizing vectorized operations that leverage pandas to perform operations on whole batches of data at once. This approach is much more efficient than processing data row by row as is typical with standard PySpark UDFs, which can significantly speed up the computation.ReferencePySpark Documentation on UDFs:https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#pandas-udfs- a-k-a-vectorized-udfs
A data scientist wants to tune a set of hyperparameters for a machine learning model. They have wrapped a Spark ML model in the objective function objective_function and they have defined the search space search_space.As a result, they have the following code block:Which of the following changes do they need to make to the above code block in order to accomplish the task?
Answer(s): A
The SparkTrials() is used to distribute trials of hyperparameter tuning across a Spark cluster. If the environment does not support Spark or if the user prefers not to use distributed computing for this purpose, switching to Trials() would be appropriate. Trials() is the standard class for managing search trials in Hyperopt but does not distribute the computation. If the user is encountering issues with SparkTrials() possibly due to an unsupported configuration or an error in the cluster setup, using Trials() can be a suitable change for running the optimization locally or in a non-distributed manner.ReferenceHyperopt documentation: http://hyperopt.github.io/hyperopt/
A machine learning engineer would like to develop a linear regression model with Spark ML to predict the price of a hotel room. They are using the Spark DataFrame train_df to train the model.The Spark DataFrame train_df has the following schema:The machine learning engineer shares the following code block:Which of the following changes does the machine learning engineer need to make to complete the task?
In Spark ML, the linear regression model expects the feature column to be a vector type. However, if the features column in the DataFrame train_df is not already in this format (such as being a column of type UDT or a non-vectorized type), the engineer needs to convert it to a vector column using a transformer like VectorAssembler. This is a critical step in preparing the data for modeling as Spark ML models require input features to be combined into a single vector column.ReferenceSpark MLlib documentation for LinearRegression: https://spark.apache.org/docs/latest/ml- classification-regression.html#linear-regression
Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?
Spark ML (Machine Learning Library) is designed specifically for handling large-scale data processing and machine learning tasks directly within Apache Spark. It provides tools and APIs for large-scale feature engineering without the need to rely on user-defined functions (UDFs) or pandas Function API, allowing for more scalable and efficient data transformations directly distributed across a Spark cluster. Unlike Keras, pandas, PyTorch, and scikit-learn, Spark ML operates natively in a distributed environment suitable for big data scenarios.
Spark MLlib documentation (Feature Engineering with Spark ML).
Post your Comments and Discuss Databricks Certified Machine Learning Associate exam prep with other Community members:
Our website is free, but we have to fight against AI bots and content theft. We're sorry for the inconvenience caused by these security measures. You can access the rest of the Certified Machine Learning Associate content, but please register or login to continue.