Course Website and Github

The term no one really defined

• What is data science? 😕

Data science is the discipline of making data useful. Ok…so what is it?

All companies claim to be data-driven, but they are different…

All companies claim to be data-driven, but they are different…

Excerpt from How Airbnb Democratizes Data Science With Data University:

Three tracks of data science

• Engineering (infrastructure and production): the process of making everything else possible

• Analysis: the process of turning raw information into insights in a fast way

• Modeling/Inference: the process of diving deeper into the data to discover the pattern we don’t easily see

(It is a group work from https://github.com/brohrer/academic_advisory/blob/master/authors.md !)

Engineering

1. Data environment: data storage, Kafka platform, Hadoop and Spark cluster etc.

2. Data management: parsing the logs, web scraping, API queries, and interrogating data streams.

3. Production: integrate model and analysis into the production system

Analysis

1. Domain knowledge

2. Exploratory analysis

3. Storytelling

Modeling

Problem to solve:

• Prediction/classification: image recognition, machine translation, spam/not_spam

• Explanation: customer segmentation, feature prioritization

• Causal inference: vaccine effectiveness, policy change

🔑 Questions

• Do we want to intervene?

• Is the cost of an error too high?

• Does the problem have a simple objective?

💡 Waffle Houses and Divorce Rate

• An example from Statistical Rethinking
• You can get the data here

##     Location WaffleHouses South MedianAgeMarriage Marriage Divorce
## 1    Alabama          128     1              25.3     20.2    12.7
## 2     Alaska            0     0              25.2     26.0    12.5
## 3    Arizona           18     0              25.8     20.3    10.8
## 4   Arkansas           41     1              24.3     26.4    13.5
## 5 California            0     0              26.8     19.1     8.0
## 6   Colorado           11     0              25.7     23.5    11.6

💡 Waffle Houses and Divorce Rate

💡 Waffle Houses and Divorce Rate

Modeling

General Process of Modeling/Analytics

Some confusions and more to come

• Data infrastructure engineer: Go,Python, AWS/Google Cloud/Azure, logstash, Kafka, and Hadoop
• Data engineer: spark/scala, python, SQL, AWS/Google Cloud/Azure, Data modeling
• BI engineer: Tableau/looker/Mode etc, data visualization, SQL, Python
• Data analyst: SQL, basic statistics, data visualization
• Data scientist: R/Python, SQL, basic + applied statistics, data visualization, experimental design
• Research scientist: R/Python, advanced statistics + experimental design, ML, research background, publications, conference contributions, algorithms
• Applied scientist: Similar to research scientist, ML algorithm design, often with an expectation of basic software engineering skills
• Machine Learning Engineer: More advanced software engineering skillset, algorithms, machine learning algorithm design, system design
• …

Three tracks of data science

Three tracks of data science

Three tracks of data science

Types of Questions (Modeling/Analytics)

• Comparison:
  • Are males more inclined to buy our products than females?
  • Are there any differences in customer satisfaction in different business districts?
  • Do soybean carrying a particular gene have higher oil content?
• Description:
  • Is the distribution of annual income normal?
  • Are there outliers?
  • What are the means of different customer segments?
• Clustering:
  • Which customers have similar product preferences?
  • Which printer performs a similar pattern to the broken ones?
  • How many different themes are there in the corpus?

Types of Questions (Modeling/Analytics)

• Classification:
  • Who is more likely to buy our product?
  • Is the borrower going to pay back?
  • Is it spam?
• Regression:
  • What will be the temperature tomorrow?
  • What is the projected net income for the next season?
  • How much inventory should we have?
• Optimization:
  • What is the best route to deliver the packages?
  • What is the optimal advertisement strategy to promote a new product?

Data Science Types v.s Needs

Data Flow

Data Science Roles

Some links

• Introduction to Data Science (still writing): the content of the workshop is in this book. Here is the [GitHub] for the book with links to code notebooks.
• Awesome-Data-Science-Materials
• The Elements of Statistical Learning
• An Introduction to Statistical Learning
• The caret package
• Hard copy books:
  • Applied Predictive Modeling
  • R for Marketing Research and Analytics
  • Statistical Rethinking