Causal Unmix
The Causal Unmix settings.
Causal Unmix splits one dataset into several, by latent regime. It is for data you suspect were generated by more than one mechanism — two populations, two experimental conditions, a process that changed partway through the file — where a single search over the pooled rows would recover neither mechanism but some average of them that corresponds to nothing.
It clusters rows on their residual signatures: each variable is regressed on a parent set, and the pattern of residuals, rather than the raw values, is what the clustering sees. Rows whose residuals behave alike are grouped. The clustering itself is a Gaussian mixture fitted by expectation-maximization, optionally with model selection over the number of components.
The output is several datasets in the Data box. You then search each one separately, and compare.
Getting to it
In a Data box, add a Causal Unmix node — it is listed under Manipulation of Data — with your dataset as its parent.
What it requires
Four conditions are enforced, and each is a hard error rather than a warning, so it is worth checking them before you start:
Exactly one dataset. It splits one dataset; it does not combine several.
Tabular data. A covariance matrix cannot be split, since there are no rows to assign.
All continuous. A dataset with discrete columns is refused.
No missing values. A dataset with missing cells is refused.
The last two conditions used to be unchecked, and a dataset that violated them produced output that looked plausible and was meaningless. They are now refused outright. If your data fail them, deal with the discrete columns or the missingness first — the Data Audit will tell you what you are facing.
Set K explicitly
This is the one piece of advice that matters most on this page.
The number of components, K, can either be set by you or chosen automatically by BIC over a
range [Kmin, Kmax]. With K fixed, the algorithm does what it should. With auto-selection,
BIC tends to choose too many components — it finds structure in the residuals that is not a
regime difference, and you get extra clusters that split one mechanism rather than separating two.
So: decide K from what you know about the data — how many populations, conditions or epochs you think are in there — and set it. Use auto-selection to explore, not to decide. If you do use it, raise the Safety margin (Auto) setting, which makes the criterion more reluctant to add a component, and look at the per-K BIC values in the diagnostics rather than taking the winner on trust.
When in doubt, run it at two or three values of K and compare the cluster sizes and the graphs you get. A regime split that is real tends to be stable across nearby K; one that is an artifact tends not to be.
The settings
Number of components. K when you set it yourself; Kmin and Kmax bound the search when Auto-select K by BIC is ticked, with Safety margin (Auto) penalizing extra components.
Residual signatures. Use parent superset regresses each variable on a shared superset of parents rather than on its own parent set, which makes the residual signatures comparable across variables; Top-M sets how many parents the superset keeps, ranked by the Score you choose — Kendall’s tau or Spearman’s rho. Robustly scale residuals (MAD) scales each residual by its median absolute deviation instead of its standard deviation, so a few extreme rows cannot dominate the clustering.
Covariance. The mixture components can use a Full covariance matrix, a Diagonal one, or Auto, which picks based on the data. Full is more expressive and needs more rows per component; diagonal is the safer choice when K is large or the sample is small.
EM. K-means restarts sets how many starting points the initialization tries, EM iters the iteration cap, and Ridge and Shrinkage regularize the component covariances — raise them if a component collapses onto too few rows. Anneal steps and Start T control deterministic annealing, which makes the fit less sensitive to its starting point.
Diagnostics. Log responsibilities, per-K BIC, and cluster sizes writes the clustering’s internals to the log. Turn this on. The cluster sizes tell you whether you have a genuine split or one large cluster and a handful of strays, and the per-K BIC values are what you need to judge an automatic choice of K.
Reading the result
The split datasets appear in the Data box, ready to search individually.
Two things to check before you believe a split:
Cluster sizes. A component holding a handful of rows is usually an artifact, not a regime. Its graph will be estimated from almost nothing whatever the search says.
Whether the regimes differ in a way this method can see. The clustering reads residual structure. Two regimes that differ in their means but share their residual structure are not what it is looking for, and two that differ only in a variable you did not include in the parent sets will not separate.
Row assignments are soft internally — each row has a responsibility for each component — but the output datasets are a hard partition, so a row near a boundary lands in one of them. The responsibilities in the diagnostics log are where you can see how confident the assignment was.