The data science team has requested assistance in accelerating queries on free form text from user reviews. The data is currently stored in Parquet with the below schema:
item_id INT, user_id INT, review_id INT, rating FLOAT, review STRING
The review column contains the full text of the review left by the user. Specifically, the data science team is looking to identify if any of 30 key words exist in this field.
A junior data engineer suggests converting this data to Delta Lake will improve query performance.
Which response to the junior data engineer s suggestion is correct?
- Delta Lake statistics are not optimized for free text fields with high cardinality.
- Text data cannot be stored with Delta Lake.
- ZORDER ON review will need to be run to see performance gains.
- The Delta log creates a term matrix for free text fields to support selective filtering.
- Delta Lake statistics are only collected on the first 4 columns in a table.
Reveal Solution Next Question