All records from an Apache Kafka producer are being ingested into a single Delta Lake table with the following schema:
key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp LONG
There are 5 unique topics being ingested. Only the "registration" topic contains Personal Identifiable Information (PII). The company wishes to restrict access to PII. The company also wishes to only retain records containing PII in this table for 14 days after initial ingestion. However, for non-PII information, it would like to retain these records indefinitely.
Which of the following solutions meets the requirements?
- All data should be deleted biweekly; Delta Lake's time travel functionality should be leveraged to maintain a history of non-PII information.
- Data should be partitioned by the registration field, allowing ACLs and delete statements to be set for the PII directory.
- Because the value field is stored as binary data, this information is not considered PII and no special precautions should be taken.
- Separate object storage containers should be specified based on the partition field, allowing isolation at the storage level.
- Data should be partitioned by the topic field, allowing ACLs and delete statements to leverage partition boundaries.
Reveal Solution Next Question