Problem Scenario 37 : ABCTECH.com has done survey on their Exam Products feedback using a web based form. With the following free text field as input in web ui.Name: StringSubscription Date: StringRating : StringAnd servey data has been saved in a file called spark9/feedback.txtChristopher|Jan 11, 2015|5Kapil|11 Jan, 2015|5Thomas|6/17/2014|5John|22-08-2013|5Mithun|2013|5Jitendra||5Write a spark program using regular expression which will filter all the valid dates and save in two separate file (good record and bad record)
Answer(s): A
Solution :Step 1: Create a file first using Hue in hdfs.Step 2: Write all valid regular expressions sysntex for checking whether records are having valid dates or not.val regl =......(\d+)\s(\w{3})(, )\s(\d{4}).......r//11 Jan, 2015 val reg2 =......(\d+)(U)(\d+)(U)(\d{4})......s II 6/17/2014 val reg3 =......(\d+)(-)(\d+)(-)(\d{4})""".r//22-08-2013 val reg4 =......(\w{3})\s(\d+)(, )\s(\d{4})......s II Jan 11, 2015Step 3: Load the file as an RDD.val feedbackRDD = sc.textFile("spark9/feedback.txt"}Step 4: As data are pipe separated , hence split the same. val feedbackSplit = feedbackRDD.map(line => line.split('|'))Step 5: Now get the valid records as well as , bad records.val validRecords = feedbackSplit.filter(x =>(reg1.pattern.matcher(x(1).trim).matches|reg2.pattern.matcher(x(1).trim).matches|reg3.patt ern.matcher(x(1).trim).matches | reg4.pattern.matcher(x(1).trim).matches)) val badRecords = feedbackSplit.filter(x =>!(reg1.pattern.matcher(x(1).trim).matches|reg2.pattern.matcher(x(1).trim).matches|reg3.pat tern.matcher(x(1).trim).matches | reg4.pattern.matcher(x(1).trim).matches))Step 6: Now convert each Array to Stringsval valid =vatidRecords.map(e => (e(0), e(1), e(2)))val bad =badRecords.map(e => (e(0), e(1), e(2)))Step 7: Save the output as a Text file and output must be written in a single tile, valid.repartition(1).saveAsTextFile("spark9/good.txt") bad.repartition(1).saveAsTextFile("sparkS7bad.txt")
Problem Scenario 25 : You have been given below comma separated employee information. That needs to be added in /home/cloudera/flumetest/in.txt file (to do tail source)sex, name, city1, alok, mumbai1, jatin, chennai1, yogesh, kolkata2, ragini, delhi2, jyotsana, pune1, valmiki, bangloreCreate a flume conf file using fastest non-durable channel, which write data in hive warehouse directory, in two separate tables called flumemaleemployee1 and flumefemaleemployee1(Create hive table as well for given data}. Please use tail source with /home/cloudera/flumetest/in.txt file.Flumemaleemployee1 : will contain only male employees data flumefemaleemployee1 :Will contain only woman employees data
Solution :Step 1: Create hive table for flumemaleemployeel and .' CREATE TABLE flumemaleemployeel(sex_type int, name string, city string )ROW FORMAT DELIMITED FIELDS TERMINATED BY ', ';CREATE TABLE flumefemaleemployeel(sex_type int, name string, city string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ', ';Step 2: Create below directory and file mkdir /home/cloudera/flumetest/ cd /home/cloudera/flumetest/Step 3: Create flume configuration file, with below configuration for source, sink and channel and save it in flume5.conf.agent.sources = tailsrcagent.channels = mem1 mem2agent.sinks = stdl std2agent.sources.tailsrc.type = execagent.sources.tailsrc.command = tail -F /home/cloudera/flumetest/in.txt agent.sources.tailsrc.batchSize = 1agent.sources.tailsrc.interceptors = i1 agent.sources.tailsrc.interceptors.i1.type = regex_extractor agent.sources.tailsrc.interceptors.il.regex = A(\\d} agent.sources.tailsrc. interceptors. M.serializers = t1 agent.sources.tailsrc. interceptors, i1.serializers.t1. name = typeagent.sources.tailsrc.selector.type = multiplexing agent.sources.tailsrc.selector.header = type agent.sources.tailsrc.selector.mapping.1 = memi agent.sources.tailsrc.selector.mapping.2 = mem2agent.sinks.std1.type = hdfsagent.sinks.stdl.channel = mem1agent.sinks.stdl.batchSize = 1agent.sinks.std1.hdfs.path = /user/hive/warehouse/flumemaleemployeei agent.sinks.stdl.rolllnterval = 0agent.sinks.stdl.hdfs.tileType = Data Streamagent.sinks.std2.type = hdfsagent.sinks.std2.channel = mem2agent.sinks.std2.batchSize = 1agent.sinks.std2.hdfs.path = /user/hi ve/warehouse/fIumefemaleemployee1 agent.sinks.std2.rolllnterval = 0agent.sinks.std2.hdfs.tileType = Data Streamagent.channels.mem1.type = memory agent.channels.meml.capacity = 100 agent.channels.mem2.type = memory agent.channels.mem2.capacity = 100agent.sources.tailsrc.channels = mem1 mem2Step 4: Run below command which will use this configuration file and append data in hdfs.Start flume service:flume-ng agent -conf /home/cloudera/flumeconf -conf-file /home/cloudera/fIumeconf/flume5.conf --name agentStep 5: Open another terminal create a file at /home/cloudera/flumetest/in.txt.Step 6: Enter below data in file and save it.l.alok.mumbai1 jatin.chennai1, yogesh, kolkata2, ragini, delhi2, jyotsana, pune1, valmiki, bangloreStep 7: Open hue and check the data is available in hive table or not.Step 8: Stop flume service by pressing ctrl+c
Problem Scenario 16 : You have been given following mysql database details as well as other info.user=retail_dbapassword=clouderadatabase=retail_dbjdbc URL = jdbc:mysql://quickstart:3306/retail_dbPlease accomplish below assignment.1. Create a table in hive as below.create table departments_hive(department_id int, department_name string);2. Now import data from mysql table departments to this hive table. Please make sure thatdata should be visible using below hive command, select" from departments_hive
Solution :Step 1: Create hive table as said.hiveshow tables;create table departments_hive(department_id int, department_name string);Step 2: The important here is, when we create a table without delimiter fields. Then default delimiter for hive is ^A (\001). Hence, while importing data we have to provide proper delimiter.sqoop import \-connect jdbc:mysql://quickstart:3306/retail_db \~username=retail_dba \-password=cloudera \--table departments \--hive-home /user/hive/warehouse \-hive-import \-hive-overwrite \--hive-table departments_hive \--fields-terminated-by '\001'Step 3: Check-the data in directory.hdfs dfs -Is /user/hive/warehouse/departments_hivehdfs dfs -cat/user/hive/warehouse/departmentshive/part'Check data in hive table.Select * from departments_hive;
Problem Scenario 68 : You have given a file as below.spark75/f ile1.txtFile contain some text. As given Belowspark75/file1.txtApache Hadoop is an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common and should be automatically handled by the frameworkThe core of Apache Hadoop consists of a storage part known as Hadoop Distributed File System (HDFS) and a processing part called MapReduce. Hadoop splits files into large blocks and distributes them across nodes in a cluster. To process data, Hadoop transfers packaged code for nodes to process in parallel based on the data that needs to be processed.his approach takes advantage of data locality nodes manipulating the data they have access to to allow the dataset to be processed faster and more efficiently than it would be in a more conventional supercomputer architecture that relies on a parallel file system where computation and data are distributed via high-speed networkingFor a slightly more complicated task, lets look into splitting up sentences from our documents into word bigrams. A bigram is pair of successive tokens in some sequence. We will look at building bigrams from the sequences of words in each sentence, and then try to find the most frequently occuring ones.The first problem is that values in each partition of our initial RDD describe lines from the file rather than sentences. Sentences may be split over multiple lines. The glom() RDD method is used to create a single entry for each document containing the list of all lines, we can then join the lines up, then resplit them into sentences using "." as the separator, using flatMap so that every object in our RDD is now a sentence.A bigram is pair of successive tokens in some sequence. Please build bigrams from the sequences of words in each sentence, and then try to find the most frequently occuring ones.
Solution :Step 1: Create all three tiles in hdfs (We will do using Hue}. However, you can first create in local filesystem and then upload it to hdfs.Step 2: The first problem is that values in each partition of our initial RDD describe lines from the file rather than sentences. Sentences may be split over multiple lines. The glom() RDD method is used to create a single entry for each document containing the list of all lines, we can then join the lines up, then resplit them into sentences using "." as the separator, using flatMap so that every object in our RDD is now a sentence. sentences = sc.textFile("spark75/file1.txt") \ .glom() \ map(lambda x: " ".join(x)) \ .flatMap(lambda x: x.spllt("."))Step 3: Now we have isolated each sentence we can split it into a list of words and extract the word bigrams from it. Our new RDD contains tuples containing the word bigram (itself a tuple containing the first and second word) as the first value and the number 1 as the second value. bigrams = sentences.map(lambda x:x.split()) \ .flatMap(lambda x: [((x[i], x[i+1]), 1)for i in range(0, len(x)-1)])Step 4: Finally we can apply the same reduceByKey and sort steps that we used in the wordcount example, to count up the bigrams and sort them in order of descending frequency. In reduceByKey the key is not an individual word but a bigram.freq_bigrams = bigrams.reduceByKey(lambda x, y:x+y)\map(lambda x:(x[1], x[0])) \sortByKey(False)freq_bigrams.take(10)
CORRECT TEXTProblem Scenario 79 : You have been given MySQL DB with following details.user=retail_dbapassword=clouderadatabase=retail_dbtable=retail_db.orderstable=retail_db.order_itemsjdbc URL = jdbc:mysql://quickstart:3306/retail_dbColumns of products table : (product_id | product categoryid | product_name | product_description | product_prtce | product_image )Please accomplish following activities.1. Copy "retaildb.products" table to hdfs in a directory p93_products2. Filter out all the empty prices3. Sort all the products based on price in both ascending as well as descending order.4. Sort all the products based on price as well as product_id in descending order.5. Use the below functions to do data ordering or ranking and fetch top 10 elements top()takeOrdered() sortByKey()
Solution :Step 1: Import Single table .sqoop import --connect jdbc:mysql://quickstart:3306/retail_db -username=retail_dba - password=cloudera -table=products -target-dir=p93_products -m 1 Note : Please check you dont have space between before or after '=' sign. Sqoop uses the MapReduce framework to copy data from RDBMS to hdfsStep 2: Read the data from one of the partition, created using above command, hadoop fs -cat p93_products/part-m-00000Step 3: Load this directory as RDD using Spark and Python (Open pyspark terminal and do following). productsRDD = sc.textFile("p93_products")Step 4: Filter empty prices, if exists#filter out empty prices linesnonemptyjines = productsRDD.filter(lambda x: len(x.split(", ")[4]) > 0)Step 5: Now sort data based on product_price in order.sortedPriceProducts=nonempty_lines.map(lambdaline:(float(line.split(", ")[4]), line.split(", ")[2] )).sortByKey()for line in sortedPriceProducts.collect(): print(line)Step 6: Now sort data based on product_price in descending order.sortedPriceProducts=nonempty_lines.map(lambda line:(float(line.split(", ")[4]), line.split(", ")[2])).sortByKey(False) for line in sortedPriceProducts.collect(): print(line)Step 7: Get highest price products name.sortedPriceProducts=nonemptyJines.map(lambda line : (float(line.split(", ")[4]), line- split(, , , , , )[2]))-sortByKey(False).take(1)print(sortedPriceProducts)Step 8: Now sort data based on product_price as well as product_id in descending order. #Dont forget to cast string #Tuple as key ((price, id), name) sortedPriceProducts=nonemptyJines.map(lambda line : ((float(line print(sortedPriceProducts)Step 9: Now sort data based on product_price as well as product_id in descending order, using top() function.#Dont forget to cast string#Tuple as key ((price, id), name)sortedPriceProducts=nonemptyJines.map(lambda line: ((float(line.s^^ print(sortedPriceProducts)Step 10: Now sort data based on product_price as ascending and product_id in ascending order, using takeOrdered{) function.#Dont forget to cast string#Tuple as key ((price, id), name) sortedPriceProducts=nonemptyJines.map(lambda line:((float(line.split(", "}[4]}, int(line.split(", "}[0]}}, line.split(", "}[2]}}.takeOrdered(10, lambda tuple :(tuple[0][0], tuple[0][1]))Step 11: Now sort data based on product_price as descending and product_id in ascending order, using takeOrdered() function.#Dont forget to cast string#Tuple as key ((price, id}, name)#Using minus(-) parameter can help you to make descending ordering , only for numeric value.sortedPrlceProducts=nonemptylines.map(lambda line:((float(line.split(", "}[4]}, int(line.split(", "}[0]}}, line.split(", "}[2]}}.takeOrdered(10, lambda tuple :(-tuple[0][0], tuple[0][1]}}
Post your Comments and Discuss Cloudera CCA175 exam prep with other Community members:
Our website is free, but we have to fight against AI bots and content theft. We're sorry for the inconvenience caused by these security measures. You can access the rest of the CCA175 content, but please register or login to continue.