Welcome to My Creative Journey

This week, we learned how to use web crawlers. Scraper is a special tool that can help us collect desired data on a website, such as video information on a video platform (including page size and duration).Through web crawlers, we can more conveniently and quickly obtain and integrate data. This is very useful for our analysis process. For example, if I want to know which movies are in the top 250 of this year's Douban movies, we can use a crawler to extract the names of each movie.

Here's the process we followed:

1. First, open the scraper tool
2. Create a new sitemap and a new selector, linking the type of data you want to collect
3. Save the selector and open the data preview to check the results

However, I believe there are ethical challenges when using web crawlers: Firstly, some data is exclusive to companies and should be respected as proprietary information. Secondly, Unrestricted use of web crawlers may result in significant consumption of server bandwidth and computing resources, leading to obstruction of normal user access.

This week, we conducted a survey in groups. We use an AI research company as a background to study the use of AI among students for developing the company's products.

In this group assignment, I learned that when conducting a survey, attention should be paid to: 1. identifying an effective research direction 2. using quantitative or qualitative scientific survey methods to design feasible survey plans 3. paying attention to ethical issues in data acquisition 4. collecting as much effective data as possible 5. using certain methods to analyze the data

Today we learned about data visualization. Extract the data from the previous course and create a graph. In addition to pie charts, bar charts, and line charts, we also encountered many different statistical charts. I also noticed during the process of plotting that not all graphs are omnipotent, and some data in pie charts or bar charts have unclear meanings.