1. Teach statistical thinking.
   • Teach statistics as an investigative process of problem-solving and decision-making.
   • Give students experience with multivariable thinking.

2. Focus on conceptual understanding.

3. Integrate real data with a context and purpose.

4. Foster active learning.

5. Use technology to explore concepts and analyze data.

6. Use assessments to improve and evaluate student learning.

The StatPREP definition:

• captured in the wild: not collected for teaching statistics
• appropriate for investigation of a particular question or set of questions
  • large enough (think: \(n \ge 1000\))
  • rich enough (think: 5 or more variables)

For the moment, put aside the important matters of context and purpose.

Instead, focus on the organization of data in a way that will contribute to

• good habits in the workplace
• ability to use technology effectively
• multivariable thinking

Discipline in presenting data encourages good work habits.

• Almost all StatPREP participants say they teach little or nothing about organizing data.
• In textbooks, teaching about data focusses on variable types, not organization.
• Organizing data properly is critically important for …
  • thinking correctly with data
  • using real data with technology

• Ironically, this is a very mathematical form of thinking, but few mathematicians know about it.
• SQL is the algebra that gets you a job
• Data base thinking \(\to\) explore and present both simple and complex data
• Fundamental to data science

unruly: (adj.) disorderly and disruptive and not amenable to discipline or control

unruly: (adj.) disorderly and disruptive and not amenable to discipline or control

Example 1

unruly: (adj.) disorderly and disruptive and not amenable to discipline or control

Example 2

1. Always in a “spreadsheet” (ie, rows and columns, electronic) fail ✔︎✔︎✔︎
2. Rows: One row per unit of observation/analysis
   • No other info (e.g. summaries, codebook info) in spreadsheet fail
   • Any subset of rows is a valid data set.
3. Columns: One column per variable fail
   • Always use column names fail
4. Always have a codebook fail
   • Rows: how cases were selected
   • Columns: units, specific questions/protocols/methods
5. Segregate data collection and data analysis
   • Never change the original data as part of the analysis

1. Always in a “spreadsheet” (ie, rows and columns) fail ✔︎✔︎✔︎
2. Rows: One row per unit of observation/analysis
3. Columns: One column per variable fail
4. Always have a codebook fail
5. Segregate data collection and data analysis

Note: This sort of tidy/rectangular data is not the only kind of data, but it is an extremely important kind of data and sufficient for introductory courses.

You need to have a way to distribute such data.

• Google spreadsheets, google forms
  • Best practices

• CSV files

• More efficient forms

One advantage of tidy data is that it is ammenable to standard, reusable data operations.

• summarize / reduce: convert entire data set to 1-row of summary values
  • in spreadsheet: often not tidy, summary formulas can be placed in untidy places
  • in R: df_stats()
• mutate / transform: add new variable
  • in spreadsheet: add a new column
  • in R: mutate()
• filter: remove/ignore rows or columns
  • in spreadsheet: “hide” data
  • in R: filter()

• select: remove/ignore columns
  • in spreadsheet: “hide” data
  • in R: select()
• arrange
  • in spreadsheet, “sort”. But how do you do this without altering the data?
  • in R: arrange()

• group: handle summarize and mutate in a groupwise way
  • in spreadsheet: pivot table
  • in R: group_by(); formulas (eg, blood_pressure ~ age + sex)
• join: bring together data from multiple sources
  • in spreadsheet “fusion tables” but will be phased out in Dec. 2019 in terms of SQL operations
• spread / gather: change unit of analysis
  • in spreadsheet pivot/unpivot
  • in R: spread(), gather() [soon to be pivot_wide(), pivot_long()]

List the many ways in which this “table” violates the conventions for effective data organization.

These are often introduced as definitions before the student has any use for them.

In stats book:

Key distinction / minor distinctions

• numerical: interval, ratio, count, measure, …
• categorical: unordered, ordered

Numerical

• quantities (with dimension and units) –
• times and dates
• counts (natural numbers)
• codes: codes represent discrete levels, not really numerical

Categorical

• “Two-sample” isn’t (usually) two samples
  • one sample, two variables
  • categorical variable identifies the group
• Summarizing with proportions doesn’t make categrical data numerical
• Fixed vs “Random”
  • Fixed: data contains all levels of interest
  • Random: data represents a subset of/sample from levels of interest