MDA621 Software Practice for Big Data Analytics Assignment Help

Assessment Details and Submission Guidelines
Trimester	T2 2024
Course Name	Master of Data Analytics (MDA)
Unit Code	MDA621
Unit Title	Software Practice for Big Data Analytics
Assessment Type	Assignment 2 (Group Assessment)
Assessment Title	Perform Data Analytics on Real-World Problems Using Amazon Web Services.
Purpose of the assessment (with ULO Mapping)	This assignment assesses the following Unit Learning Outcomes; students should be able to demonstrate their achievements in them. c. Select the tools in the chosen software stack to design and program the big data analytics platform; d. Relate the concept and use of visualization to big data analytics; e. Develop and appraise big data platforms for predictive analytics in complex real world domains.
Weight	25%
Total Marks	100 Marks
Word limit	700-1200 Words
Due Date	29 Sep 2024, 23:59 – Week 11 Sunday
Submission Guidelines	GENERATIVE AI TOOLS MAY BE USED WITH PRIOR PERMISSION Students are permitted to use advanced automated tools for this formative assessment only for understanding, learning and research purposes. Using Gen AI tools to write assignments for you, will be considered as Academic Misconduct, and it will be penalized. If students are using any of the information from Gen AI, then you must cite or attribute the use of the Gen AI in their assessment. Group Submit (only the group leader): • Name your final report “MDA621_T2_2024_Assigment2_GroupID.docx” • Results, including codes and commands, in a MS Word file. • A maximum 10-minute recorded video (mp4 format) with all members participating in presenting your analytics solutions. It is recommended that the recorded video be uploaded to YouTube, and then include the URL link towards the end of the final report. Please do not upload video directly onto Moodle. Individual Submit (every student): • Complete the “Peer Evaluation” table by commenting on contributions and participation by your team members. Please include your team member’s name in the evaluation. • The assignment must be in MS Word format, 1.5 spacing, 11-pt Calibri (Body) font and 2.5 cm margins on all four sides of your page with appropriate section headings. • Reference sources must be cited in the text of the report and listed appropriately at the end in a reference list using either the IEEE or the APA referencing style.

Extension

• If an extension of time to submit work is required, a Special Consideration Application must be submitted directly on AMS. You must submit this application three working days prior to the due date of the assignment. Further information is available at

Academic

Misconduct

Academic Misconduct is a serious offense. Depending on the seriousness of the case, penalties can vary from a written warning or zero marks to exclusion from the course or rescinding the degree. Students should make themselves familiar with the full policy and procedure available at:.

For further information, please refer to the Academic Integrity Section in your Unit Description.

Description

In this group assignment, you will explore various aspects of big data analysis and manipulation using the Hadoop ecosystem, specifically focusing on Pig Latin and Hive Query Language (Hive QL). The primary objective of this assignment is to gain practical experience in processing large-scale data sets and understanding the differences in data trends over time. You will be working with two distinct data sets: patent data and sale data related to patent applications and sale transactions.

For the first part of the assignment, you will use patent data from the United States (US) and abroad to analyze the total patent applications applied and granted each year. This will involve executing patent files, creating directories on a Hadoop cluster, and running the Pig Latin program to count the total number of patent applications applied and granted yearly. This analysis will help you understand the development of analytics reports on patent data over a decade.

In the second part of the assignment, you will work with sales data using Hive QL commands. The primary goal here is to analyze product sale transactions by performing various data operations such as uploading files to HDFS, joining multiple data sets, grouping data, and performing calculations such as listing customers, products and supplier details. You will also filter and display the expensive products and the average price of all products.

By the end of this assignment, you will have acquired hands-on experience in leveraging the Hadoop ecosystem to analyze and manipulate large-scale data sets efficiently. This will serve as a foundation for further exploration in the field of big data analysis and processing.

Note: As a starting point, it is essential to apply the knowledge gained from previous lessons to create a cluster on Amazon Web Services (AWS) and set it up properly to execute all required files. Including, setting up the EC2 and EMR cluster on AWS, SSH to connect to the cluster, and setting up the File Transfer Protocol. These initial steps lay the foundation for successfully managing and processing large-scale data sets using the Hadoop ecosystem on a cloud-based platform.

Your Tasks

To complete Assignment 2, which comprises two main parts, your team will follow the steps outlined in the two questions below to perform data processing and analysis tasks using the Hadoop ecosystem, Pig Latin and Hive QL. The primary focus will be working with datasets related to patent data and sale data, allowing for hands-on experience in managing and processing large-scale information efficiently.

Part I: Download the US_patent.csv data from the Assignment 2 folder on Moodle. It is a comma separated value (CSV) file. The file includes a list of US patents applied and granted between 1965 and 2020. The column “Total Patent Applications” in the file presents the total applied applications, and “Total Patent Grants” contains information about total granted applications.

[40 Marks]

For Part I, using Pig Latin commands and Tableau to perform the following tasks:

1. Upload the files to HDFS.

2. Create directories on the cluster and name the directory Patent.

3. Load the file US_patent to the new directory.

4. Write a Pig program to find the total number of patents applied each year.

5. Write a Pig program to find the total number of patents granted each year.

6. Observe and compare the applied rate and granted rate from 1965 and 2020.

7. Using Tableau Software, visualize the results in a suitable manner. Choose the format that you find most appropriate.

8. In 350 words, summarize your understanding of the changing trends and accepted rates of patent applications.

Part II: Download the saledata.zip data from Assignment 2 folder on Moodle. This file is compressed, and once you unzip it, you will find five tab-delimited text formats files:

[50 Marks]

• customer (which contains information about customer ID, name and zip code)

• product (which contains information about product ID, name, price, supplier ID and product category)

• product sales(which contain information about product id, corresponding sale transaction id and no of items sold)

• sales (which contain information about sale ID, customer ID, store ID and transaction year) • supplier (which contains information about supplier ID and supplier name)

For Part II, using Hive QL commands to perform the following operations:

1. Upload the five files to HDFS

2. Create a directory on the cluster and name the directory Sales.

3. Create a database named sales_db and create tables where the above files can be loaded.

4. Display the supplier id and supplier name for all suppliers.

5. Display the customer’s name and customer zip for all customers.

6. Display the product id, product name, product price, and supplier name for all products.

7. Filter and display the product id, product name, and product price for products with a product price of $500 or lower.

8. Filter and display the customer’s name and sale year for sales involving a customer buying more than two products.

9. Filter the top product id and product name based on product price.

10. Calculate the average price of all products.

11. In 350 words, summarize your findings.

Submission

The group must submit the followings (only group leader submits on behalf of the whole group): • A final report should detail how your team has carried out each step of the data analytics process explained above, any assumptions made in the analysis, any limitations in your models, and comparison of your findings. The names of team members must be included on the cover page. • Include all Python code required for processing data analysis.

• A maximum 10-minute recorded video (.mp4 format) with all members participating in presenting this group Assignment 2. You can use PowerPoint slides in your presentation. Each student must additionally submit:

• “Peer Evaluation” table by commenting on contributions and participation by your team members. Please include your team member’s name in the evaluation.

Marking Criteria for the Assignment 2

Sections	Description of the section	Marks
Part I	• You will use US patent data from 1965 and 2020 to analyze the applied and granted applications over the decade. You will use a Pig Latin program to count the applications for each year. It will involve creating directories on a Hadoop cluster and running the Pig Latin program. Finally, visualize the results using Tableau Software.	40
Part II	• You will work with sales data using HiveQL commands. The primary goal here is to analyze sale transactions by performing various data operations such as uploading files to HDFS, joining two data sets, grouping data, and performing calculations such as total sales and customer buying patterns. You will also filter and display the top product with the highest price.	50
Video Presentation	• “Peer Evaluation” table by commenting on contributions and participation by your team members.	10
Total Marks		100