Skip to contents

Lineage and Rebuilds

Source: vignettes/lineage-rebuilds.Rmd
lineage-rebuilds.Rmd
# Use development build when interactive *and* explicitly enabled via env var.
dev_mode <- (Sys.getenv("DEV_VIGNETTES", "false") == "true")

if (dev_mode && requireNamespace("pkgload", quietly = TRUE)) {
  cli::cli_inform("loading with {.pkg pkgload}")
  pkgload::load_all()
} else {
  # fall back to the installed package (the path CRAN, CI, and pkgdown take)
  cli::cli_inform("loading with {.pkg library}")
  library(stamp)
}
## loading with library

This vignette shows how stamp captures lineage (parents → children), how to detect staleness, and how to plan & rebuild downstream artifacts in level order.

You’ll use:

Strict vs. Propagate Strict checks actual mismatch vs the parents’ current latest versions. Propagate simulates change pushing downstream (A changes ⇒ schedule B ⇒ schedule C, etc.).

A few implementation notes that clarify behavior used by the examples below:

  • Committed parents (the parents.json file inside a version snapshot) are the authoritative record for lineage when present. They are captured at commit time and are used for reproducible, recursive lineage walking.
  • Sidecar parents (the parents field in the sidecar metadata written next to the artifact file) are a lightweight convenience that let you inspect first-level lineage before a snapshot is committed. st_lineage() falls back to sidecar parents only for the immediate parent level when no snapshot is available; recursive walking beyond level 1 requires snapshot-backed parents.

This design keeps interactive workflows fast (sidecars are quick) while preserving reproducible history once snapshots are written.

Setup a tiny graph A → B(A) → C(B) 

st_opts_reset()
st_opts(
  versioning = "content",
  code_hash = TRUE,
  store_file_hash = TRUE,
  verify_on_load = TRUE,
  meta_format = "both"
)
## ✔ stamp options updated
##   versioning = "content", code_hash = "TRUE", store_file_hash = "TRUE",
##   verify_on_load = "TRUE", meta_format = "both"
root <- tempdir()
st_init(root)
## ✔ stamp initialized
##   root: /tmp/RtmpPqoLT7
##   state: /tmp/RtmpPqoLT7/.stamp
# A
pA <- fs::path(root, "A.qs")
xA <- data.frame(a = 1:3)
st_save(xA, pA, code = function(z) z)
## ✔ Saved [qs] → /tmp/RtmpPqoLT7/A.qs @ version
## 95d472cafdc64ac7
# B depends on A
pB <- fs::path(root, "B.qs")
xB <- transform(xA, b = a * 2)
st_save(
  xB,
  pB,
  code = function(z) z,
  parents = list(list(path = pA, version_id = st_latest(pA)))
)
## ✔ Saved [qs] → /tmp/RtmpPqoLT7/B.qs @ version
## c15db8dc94ca21f4
# C depends on B
pC <- fs::path(root, "C.qs")
xC <- transform(xB, c = b + 1L)
st_save(
  xC,
  pC,
  code = function(z) z,
  parents = list(list(path = pB, version_id = st_latest(pB)))
)
## ✔ Saved [qs] → /tmp/RtmpPqoLT7/C.qs @ version
## 9450641f526cadb3

Note: after these saves each artifact has a sidecar (in stmeta/) and snapshots under .stamp/versions/.

Committed parents vs sidecar example

This short interactive example shows the difference between committed parents (stored inside the version snapshot) and the light-weight sidecar parents next to the artifact file. We deliberately remove the committed snapshot for B to simulate a case where only the sidecar remains; st_lineage() will still return immediate parents (level 1) by reading the sidecar, but recursive walking beyond level 1 only uses snapshot-backed parents.

# Remove committed snapshot for B to simulate a no-snapshot state
vdir_b <- stamp:::.st_version_dir(pB, st_latest(pB))
if (fs::dir_exists(vdir_b)) {
  fs::dir_delete(vdir_b)
}

# Now st_info will show snapshot_dir = NA but sidecar present
st_info(pB)
## $sidecar
## $sidecar$path
## [1] "/tmp/RtmpPqoLT7/B.qs"
## 
## $sidecar$format
## [1] "qs"
## 
## $sidecar$created_at
## [1] "2025-12-22T11:01:00.241735Z"
## 
## $sidecar$size_bytes
## [1] 148
## 
## $sidecar$content_hash
## [1] "bba6fe40c1133df8"
## 
## $sidecar$code_hash
## [1] "488e8fa49c740261"
## 
## $sidecar$file_hash
## [1] "aedd22253abcd70c"
## 
## $sidecar$code_label
## NULL
## 
## $sidecar$parents
##                   path       version_id
## 1 /tmp/RtmpPqoLT7/A.qs 95d472cafdc64ac7
## 
## $sidecar$attrs
## list()
## 
## 
## $catalog
## $catalog$latest_version_id
## [1] "c15db8dc94ca21f4"
## 
## $catalog$n_versions
## [1] 1
## 
## 
## $snapshot_dir
## [1] NA
## 
## $parents
##                   path       version_id
## 1 /tmp/RtmpPqoLT7/A.qs 95d472cafdc64ac7
# st_lineage will fall back to the sidecar for immediate parents (level 1)
st_lineage(pB, depth = 1)
##   level           child_path    child_version          parent_path
## 1     1 /tmp/RtmpPqoLT7/B.qs c15db8dc94ca21f4 /tmp/RtmpPqoLT7/A.qs
##     parent_version
## 1 95d472cafdc64ac7
# But recursive lineage (depth > 1) will only follow snapshot-backed parents
# (no recursive walk available once snapshots are missing)
st_lineage(pB, depth = 2)
##   level           child_path    child_version          parent_path
## 1     1 /tmp/RtmpPqoLT7/B.qs c15db8dc94ca21f4 /tmp/RtmpPqoLT7/A.qs
##     parent_version
## 1 95d472cafdc64ac7

Inspect lineage 

# Immediate children of A (depth 1)
st_children(pA, depth = 1)
## [1] level          child_path     child_version  parent_path    parent_version
## <0 rows> (or 0-length row.names)
# Full lineage (parents of an artifact)
st_lineage(pC, depth = Inf)
##   level           child_path    child_version          parent_path
## 1     1 /tmp/RtmpPqoLT7/C.qs 9450641f526cadb3 /tmp/RtmpPqoLT7/B.qs
##     parent_version
## 1 c15db8dc94ca21f4

Make a change upstream & detect staleness 

# Change A → new version
xA2 <- transform(xA, a = a + 10L)
st_save(xA2, pA, code = function(z) z)
## ✔ Saved [qs] → /tmp/RtmpPqoLT7/A.qs @ version
## 13eda2d427672bb9
# Strict staleness
st_is_stale(pB) # TRUE (B's recorded A version is now old)
## [1] FALSE
st_is_stale(pC) # FALSE (C points to B, which hasn't changed yet)
## [1] FALSE

Plan rebuilds

Two strategies:

  • strict: only items whose recorded parent IDs differ from parents’ current latest.
  • propagate: assume targets will change and plan descendants in BFS layers.
# Strict: only B right now
plan_strict <- st_plan_rebuild(pA, depth = Inf, mode = "strict")
plan_strict
## [1] level                 path                  reason               
## [4] latest_version_before
## <0 rows> (or 0-length row.names)
# Propagate: includes B (level 1) and C (level 2)
plan <- st_plan_rebuild(pA, depth = Inf, mode = "propagate")
plan
## [1] level                 path                  reason               
## [4] latest_version_before
## <0 rows> (or 0-length row.names)

Register builders and rebuild in level order

Builders are tiny functions that produce an artifact from its parents. They receive (path, parents) and return a list with at least x = <object>.

# Clear any previous registry
st_clear_builders()
## ✔ Cleared all registered builders
# Register a builder for B: rebuild from A's committed version
st_register_builder(pB, function(path, parents) {
  # parents is list(list(path=..., version_id=...))
  A <- st_load_version(parents[[1]]$path, parents[[1]]$version_id)
  list(
    x = transform(A, b = a * 2),
    code = function(z) z,
    code_label = "B <- A * 2"
  )
})
## ✔ Registered builder for /tmp/RtmpPqoLT7/B.qs
## (default)
# Register a builder for C: rebuild from B's committed version
st_register_builder(pC, function(path, parents) {
  B <- st_load_version(parents[[1]]$path, parents[[1]]$version_id)
  list(
    x = transform(B, c = b + 1L),
    code = function(z) z,
    code_label = "C <- B + 1"
  )
})
## ✔ Registered builder for /tmp/RtmpPqoLT7/C.qs
## (default)
# Dry run first (uses registered builders found by st_rebuild when rebuild_fun is NULL)
st_rebuild(plan, dry_run = TRUE)
## ✔ Nothing to rebuild (empty plan).
# Now actually rebuild (will use registered builders)
res <- st_rebuild(plan, dry_run = FALSE)
## ✔ Nothing to rebuild (empty plan).
res
## [1] level                 path                  reason               
## [4] latest_version_before status                version_id           
## [7] msg                  
## <0 rows> (or 0-length row.names)

After rebuilding B, C becomes strictly stale if B changes again later. You can re-plan from B to keep propagating:

## [1] FALSE
## [1] FALSE
st_plan_rebuild(pB, depth = Inf, mode = "propagate")
##   level                 path           reason latest_version_before
## 1     1 /tmp/RtmpPqoLT7/C.qs upstream_changed      9450641f526cadb3

Inspect snapshots on disk 

vroot <- stamp:::.st_versions_root()
fs::dir_tree(vroot, recurse = TRUE, all = TRUE)
## /tmp/RtmpPqoLT7/.stamp/versions
## ├── A.qs
## │   ├── 13eda2d427672bb9
## │   │   ├── artifact
## │   │   ├── sidecar.json
## │   │   └── sidecar.qs2
## │   └── 95d472cafdc64ac7
## │       ├── artifact
## │       ├── sidecar.json
## │       └── sidecar.qs2
## ├── B.qs
## └── C.qs
##     └── 9450641f526cadb3
##         ├── artifact
##         ├── parents.json
##         ├── sidecar.json
##         └── sidecar.qs2

Tip: st_info(path) summarizes sidecar, catalog status, and the latest snapshot dir, plus parsed parents.json from the committed snapshot (if present).

## $sidecar
## $sidecar$path
## [1] "/tmp/RtmpPqoLT7/C.qs"
## 
## $sidecar$format
## [1] "qs"
## 
## $sidecar$created_at
## [1] "2025-12-22T11:01:00.296869Z"
## 
## $sidecar$size_bytes
## [1] 159
## 
## $sidecar$content_hash
## [1] "28e3e19ddcccd8c6"
## 
## $sidecar$code_hash
## [1] "488e8fa49c740261"
## 
## $sidecar$file_hash
## [1] "4286965c99aa6527"
## 
## $sidecar$code_label
## NULL
## 
## $sidecar$parents
##                   path       version_id
## 1 /tmp/RtmpPqoLT7/B.qs c15db8dc94ca21f4
## 
## $sidecar$attrs
## list()
## 
## 
## $catalog
## $catalog$latest_version_id
## [1] "9450641f526cadb3"
## 
## $catalog$n_versions
## [1] 1
## 
## 
## $snapshot_dir
## /tmp/RtmpPqoLT7/.stamp/versions/C.qs/9450641f526cadb3
## 
## $parents
## $parents[[1]]
## $parents[[1]]$path
## [1] "/tmp/RtmpPqoLT7/B.qs"
## 
## $parents[[1]]$version_id
## [1] "c15db8dc94ca21f4"

Takeaways

  • Use strict staleness to detect objective mismatches.
  • Use propagate planning to build a full downstream schedule.
  • Keep builders small, pure, and deterministic; they make rebuilds trivial.