na.omit, imputation, duplicated

◈ 10 cards

With order 1005's amount blank, na.omit(orders) has 11 rows — the whole row goes; complete.cases() is the TRUE/FALSE behind it. Imputing with median(orders$amount, na.rm = TRUE) fills 420 and must be recorded in the dictionary. A pasted-twice order is found by duplicated(orders$order_id) and removed with orders[!duplicated(orders$order_id), ]; unique(orders) needs every column equal.

Drop the row, or fill the cell

Module 5 gave two answers to a blank: leave the row out, or fill it with a defensible value and say so. R has one call for each. Blank order 1005's amount and count:

> ona$amount[ona$order_id == 1005] <- NA
> colSums(is.na(ona))
  order_id       prov    channel     amount      units order_date 
         0          0          0          1          0          0 
> nrow(na.omit(ona))
[1] 11

na.omit() drops every row with an NA in any column — not the cell, not the numeric columns, the row. One blank amount costs the whole order, its province and its units with it. complete.cases() is the logical vector underneath — TRUE for a row with nothing missing — and it goes in the rows slot when you want to choose columns first:

> complete.cases(ona)
 [1]  TRUE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
> sum(complete.cases(ona))
[1] 11

Worked example — impute with the median, and write it down

Eleven orders is a thin quarter; the analyst keeps the row and fills the amount with the median of the other eleven:

> median(ona$amount)
[1] NA
> median(ona$amount, na.rm = TRUE)
[1] 420
> ona$amount[is.na(ona$amount)] <- median(ona$amount, na.rm = TRUE)
> ona$amount[ona$order_id == 1005]
[1] 420

Three things in that block. median() of a column with an NA is NA (Module 9: na.rm = TRUE is opt-in). The assignment targets only the cells where is.na() is TRUE — a filtered assignment, the same rows slot as Module 11. And the median, not the mean: the mean of the other eleven is dragged by 980 and 760; the median 420 is the typical order. The real 1005 was $1,320 — no imputation recovers that, which is why the data dictionary must say amount for 1005 imputed as the median (420) on 2025-04-02 before anyone reads the province totals.

The same order twice

A March export was pasted on top of a February one and order 1004 appears twice. duplicated() marks the second and later copies of a value:

> dup <- rbind(orders, orders[4, ])
> nrow(dup)
[1] 13
> duplicated(dup$order_id)
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[13]  TRUE
> nrow(dup[!duplicated(dup$order_id), ])
[1] 12

The first 1004 is kept, the thirteenth row goes — !duplicated() in the rows slot is keep the first of each key. The key is order_id because the data dictionary says one row per order (Module 3's primary key). unique(dup) also gives 12 here, but only because the copy is identical in every column. Change one cell — the pasted 1004 says 255 instead of 250 — and the two disagree:

> dup2 <- dup
> dup2$amount[13] <- 255
> nrow(unique(dup2))
[1] 13
> nrow(dup2[!duplicated(dup2$order_id), ])
[1] 12
> dup2[duplicated(dup2$order_id), ]
   order_id prov channel amount units order_date
41     1004   BC  Online    255     2 2025-01-21

The row name is 41, not 13: rbind() kept the pasted row's own name, 4, and made it unique by appending a 1. A row name is a label carried with the row, never its position (Module 10) — the thirteenth row of dup2 is the one named 41. unique() needs every column equal; duplicated() on the key finds the same order whatever the other cells say — and the third line shows which row it is about to drop, so the analyst can decide which 1004 is right instead of keeping whichever came first.

Type the three, then read the imputed value

CRISP-DM: dropping, imputing and de-duplicating are all data preparation → clean data — and each is a decision the dictionary records.