Chapter 5: Research Provenance and Evidence Pipeline

In Chapter 4: Immutable Case, Job, Attempt, and Result Lifecycle, we followed one simulation from a frozen Case to a verified Result.

Now we will connect many verified Results into a larger research workflow.

Imagine comparing several shielding designs:

1. Define a research question.
2. Group Cases and Jobs into a Study.
3. Run replicas for each Case.
4. Recompute metrics from verified Results.
5. Decide which members meet the inclusion criteria.
6. Build a Dataset from accepted evidence.
7. Train and evaluate a Model.
8. Create Predictions.
9. Verify Predictions against new PHITS Results.

This is the Research Provenance and Evidence Pipeline.

The main idea

Research provenance means keeping a reliable history of where every scientific artifact came from.

A useful analogy is a chain of custody:

Study
  ↓
Assessment
  ↓
Dataset
  ↓
Model
  ↓
Prediction
  ↓
Prediction Verification

Each step should answer:

“What earlier record supports this artifact?”

The pipeline also checks that the earlier record is still valid. If a Result changes, an Assessment may no longer be trustworthy. If an Assessment changes, a Dataset may no longer be trustworthy either.

flowchart LR
    A[Verified Results] --> B[Study]
    B --> C[Assessment]
    C --> D[Dataset]
    D --> E[Model]
    E --> F[Prediction]
    F --> G[Prediction Verification]

What problem does this solve?

Without provenance, a machine-learning result might only say:

model-final.bin
accuracy = 0.94

That is not enough. We also need to know:

• Which Study supplied the samples?
• Which Assessment accepted those samples?
• Which Cases and Results were used?
• Which features and units were used?
• Which model was selected?
• Which PHITS Result verified the prediction?

RAMAL-EBX stores these links explicitly.

The main research records

RAMAL-EBX uses several research-level records.

Record	Purpose
Study	Defines the research question and members
Assessment	Recomputes Result metrics and applies criteria
Dataset	Stores model-ready samples and their source Study
Model	Stores a finalized model linked to a Dataset
Prediction	Stores a model output and its input features
Prediction Verification	Compares a Prediction with a real PHITS Result

The records form a dependency chain:

Study → Assessment → Dataset → Model → Prediction → Verification

A later record should not silently invent its own history.

A beginner’s example

Suppose we want to study this question:

How does shielding geometry affect the reportable dose field?

We have three Cases:

case-reference
case-thin-shield
case-thick-shield

Each Case has one or more Jobs and verified Results.

The Study groups these members:

Study: shielding-comparison
Question: How does shielding geometry affect reportable dose?
Members:
    reference Case
    thin-shield Case
    thick-shield Case

The Study does not contain the dose values itself. It identifies which simulation records should be evaluated.

Studies define a research question

A Study_Document contains the scientific grouping:

Study_Document :: struct {
    project_id: string
    study_id: string
    display_name: string
    question: string
    members: [dynamic]Study_Member
}

It also stores:

• Evaluation domain.
• Evaluation regions.
• Inclusion criteria.
• Optional reference Case.
• Creation timestamp.

The question explains why the Study exists.

For example:

question:
How does shielding thickness affect dose in the evaluation region?

This is useful to people reading the record later.

Study members

Each member connects a Study to a Case and Job.

Study_Member :: struct {
    member_id: string
    case_id: string
    job_id: string
    role: string
    replica_index: i32
}

A member might look like this:

member_id:     member-0002
case_id:       case-thick-shield
job_id:        job-003
role:          sample
replica_index: 0

The member does not directly claim that its Result is valid. It identifies the Case and Job whose Result should be searched for later.

Reference and sample members

A Study may contain a reference Case:

role: reference

Other members can be samples:

role: sample

The reference Case must be declared consistently:

reference_case_id = case-reference

RAMAL-EBX checks that:

• The reference Case appears in the member list.
• It has the reference role.
• Other Cases do not incorrectly use that role.
• Member IDs follow the expected order.
• A Case and Job pair is not duplicated.

This prevents a Study from having ambiguous membership.

Evaluation domains and regions

A Study defines where Results should be examined.

The evaluation domain is a three-dimensional grid:

Study_Evaluation_Domain :: struct {
    min_cm: [3]f32
    max_cm: [3]f32
    mesh_count: [3]i32
}

For example:

min_cm:     {-50, -50, -50}
max_cm:     { 50,  50,  50}
mesh_count: {100, 100, 100}

A region is a smaller box inside that domain:

Study_Evaluation_Region :: struct {
    region_id: string
    display_name: string
    min_cm: [3]f32
    max_cm: [3]f32
}

A region might represent:

Detector region
x: -5 to 5
y: 10 to 20
z: 40 to 50

The domain and regions must be valid. A region cannot extend outside the evaluation domain.

Inclusion criteria

Inclusion criteria say which evidence is reportable.

Study_Inclusion_Criteria :: struct {
    max_relative_error: f32
    min_reportable_fraction: f32
    min_reportable_voxel_count: i32
}

For example:

maximum relative error: 0.10
minimum reportable fraction: 0.80
minimum reportable voxels: 100

This means:

A member qualifies only if:
    enough voxels have positive dose,
    enough voxels have acceptable uncertainty,
    and the reportable fraction is high enough.

The Study stores these rules so that the assessment can repeat them later.

Creating a Study

A Study is created from existing Cases.

request := Study_Creation_Request{
    display_name = "Shielding Comparison",
    question = "How does thickness affect dose?",
    case_ids = {"case-reference", "case-thick"},
    replica_count = 2,
}

This request describes the intended Study. RAMAL-EBX then checks each Case.

The Study is created through the Project:

study_create(&app.project, request)

The result is a Study record and Jobs for its members.

The Jobs receive deterministic seeds based on their member positions. This makes replicas predictable while keeping their run identities separate.

What Study creation checks

Before creating the Study, RAMAL-EBX verifies:

• The Project is loaded.
• Case IDs are valid and unique.
• The requested replica count is positive.
• Histories and batches are positive.
• Seeds are valid.
• Every Case has a valid dose tally.
• Every Case uses the same evaluation grid.
• Every Case has a compatible Result contract.
• The reference Case is present when declared.

This is important because a Study needs comparable members.

sequenceDiagram
    participant User
    participant Study
    participant Cases
    participant Jobs
    participant Disk

    User->>Study: Create Study request
    Study->>Cases: Validate Cases and contracts
    Cases-->>Study: Compatible Cases
    Study->>Jobs: Create member Jobs
    Study->>Disk: Write Study record
    Disk-->>User: Study is available

Study storage

A Study is stored inside the Project:

studies/shielding-comparison/
└── shielding-comparison.ramal-study

The record includes the Project ID:

{
  "project_id": "alpha-study",
  "study_id": "shielding-comparison",
  "question": "How does thickness affect dose?"
}

When RAMAL-EBX reads it, the Project ID and Study ID must match the requested location.

An invalid or mismatched Study is skipped by the Study catalog.

Running a Study

A Study can run its members sequentially.

The transient state is stored in Study_Run_State:

Study_Run_State :: struct {
    study_id: string
    current_member_id: string
    member_index: int
    completed_count: int
    succeeded_count: int
    failed_count: int
}

The run starts from the first member:

study_run_start(&app, "shielding-comparison")

RAMAL-EBX sends each member through the normal Case runner. It does not create a separate simulation system.

The flow is:

Study member
    ↓
Case and Job
    ↓
Attempt
    ↓
Verified Result or failure
    ↓
Next member

This reuses the lifecycle from Immutable Case, Job, Attempt, and Result Lifecycle.

Study execution state

A Study run reports counts such as:

2/4 succeeded
1 failed
1 stopped

The Study run is temporary application state. It tells the interface what is happening now.

The Study document itself remains the durable definition of the research question and members.

This is similar to the difference between:

• A scheduled laboratory procedure.
• The permanent written research protocol.

Assessment: recomputing evidence

After Results exist, an Assessment examines them.

An Assessment does not blindly trust previously stored metrics. It loads the Result again and recomputes the metrics from its artifacts.

Assessment_Member :: struct {
    member_id: string
    case_id: string
    job_id: string
    result_id: string
    status: string
}

Possible statuses include:

missing-result
invalid-result
ambiguous-result
failed-job
does-not-meet-criteria
meets-criteria

This lets the Assessment explain why each member was accepted or rejected.

Creating an Assessment

An Assessment is created from a Study:

assessment_create(
    &app.project,
    "shielding-comparison",
)

For each Study member, RAMAL-EBX searches the member’s Job directory for Result records.

The member can have:

• No Result.
• One valid Result.
• Multiple Results.
• Invalid Result files.
• A Result that fails current verification.

Only one unambiguous, verified Result can become evidence.

Assessment creation flow

The process is:

flowchart TD
    A[Read Study] --> B[Find member Results]
    B --> C{Exactly one valid Result?}
    C -->|No Result| D[missing-result]
    C -->|Multiple| E[ambiguous-result]
    C -->|Invalid| F[invalid-result]
    C -->|Yes| G[Recompute metrics]
    G --> H[Apply inclusion criteria]
    H --> I[Store Assessment]

The Assessment is therefore a decision record based on current evidence.

Recomputing metrics from a Result

For a verified Result, RAMAL-EBX loads:

• Dose VTK data.
• Uncertainty VTK data.
• The Case Result contract.
• The Study evaluation domain.
• The Study evaluation regions.

It then checks that the Result grid matches the Study grid.

if !assessment_field_grid_matches(&field, study.domain) {
    return false
}

A mismatch means the Result cannot be compared fairly with the other Study members.

Counting reportable voxels

For every voxel inside a region, RAMAL-EBX checks the dose and relative error.

If dose > 0:
    count it as positive
    if relative error is acceptable:
        count it as reportable

The resulting metrics include:

Assessment_Region_Metrics :: struct {
    voxel_count: i32
    positive_scored_count: i32
    reportable_count: i32
    reportable_fraction: f32
    maximum_reportable_dose: f32
    maximum_reportable_rse: f32
    meets_criteria: bool
}

The reportable fraction is calculated as:

reportable fraction =
    reportable voxels / all voxels in the region

Applying the criteria

The criteria are applied directly to the recomputed metrics:

metrics.meets_criteria =
    metrics.reportable_count >= policy.min_reportable_voxel_count &&
    metrics.reportable_fraction >= policy.min_reportable_fraction

If a region report is available, RAMAL-EBX also checks the region mean and its relative error.

The member meets the Study criteria only if all required regions meet their criteria.

Example assessment result

Suppose a region contains:

voxel count:          1,000
positive voxels:        950
reportable voxels:      900
reportable fraction:  0.90
maximum relative error: 0.10

With these criteria:

minimum reportable voxels: 800
minimum fraction:          0.80

the region qualifies.

The Assessment might store:

member-0002
status: meets-criteria
result: result-0004

If only 600 voxels were reportable, the status would be:

does-not-meet-criteria

Assessment integrity

When an Assessment is loaded, RAMAL-EBX can recompute its evidence again.

It compares stored and recomputed values:

Stored reportable count
    versus
Current reportable count

It checks:

• Region IDs.
• Voxel counts.
• Reportable counts.
• Fractions.
• Maximum dose.
• Maximum relative error.
• Region means.
• Criteria decisions.

If a stored value no longer matches, the Assessment is rejected.

This protects against edited or stale assessment files.

Assessment storage

Assessments are stored below their Study:

studies/shielding-comparison/assessments/
└── assessment-0001.ramal-study-assessment

The Assessment includes:

project_id
study_id
assessment_id
members
status

The Study and Assessment IDs must agree.

The Assessment also needs the same members as the Study. A missing or reordered member is an integrity error.

Datasets: preparing model input

A Dataset converts assessed research evidence into model-ready samples.

A Dataset stores a reference back to its Study and Assessment:

Dataset_Study_Reference :: struct {
    study_id: string
    study_path: string
    assessment_id: string
    assessment_path: string
}

The Dataset also stores:

• Feature definitions.
• Partitioning method.
• Target summary.
• Samples.
• Sample splits.

A Dataset is not just a table of numbers. It is a table plus its scientific origin.

Dataset samples

A sample identifies its Study member:

Dataset_Sample :: struct {
    configuration_id: string
    member_id: string
    split: string
    feature_values: []f32
}

For example:

configuration_id: thick-shield
member_id:        member-0002
split:            train
feature_values:   [20, 1]

The feature values must match the declared feature definitions.

Feature definitions

A feature describes one input value.

ML_Feature_Definition :: struct {
    feature_id: string
    display_name: string
    kind: string
    unit: string
    minimum: f32
    maximum: f32
}

A continuous feature might be:

feature_id: thickness
kind:       continuous
unit:       cm
minimum:    1
maximum:    100

A binary feature might describe a category:

feature_id: shielding-material
kind:       binary
unit:       category
categories: concrete, steel

RAMAL-EBX checks that feature IDs are unique and that values stay inside their declared ranges.

Dataset partitions

Samples are divided into:

train
validation
final-test

A valid Dataset needs enough samples for each split.

If a paired binary feature is used, both categories must remain together in the same split group. This prevents related samples from leaking across training and testing.

The Dataset records the method:

Dataset_Partitioning_Method :: struct {
    algorithm: string
    seed: i32
    paired_feature_id: string
}

This makes the split reproducible.

Dataset provenance

A valid Dataset must point to canonical paths:

Study path:
studies/shielding-comparison/shielding-comparison.ramal-study

Assessment path:
studies/shielding-comparison/assessments/assessment-0001.ramal-study-assessment

When the Research index loads the Dataset, it checks that:

• The referenced Study exists.
• The Study is valid.
• The Assessment path is canonical.
• The Dataset has enough samples.
• Coverage values are valid.
• Feature definitions are valid.

If the Study is missing or invalid, the Dataset is rejected.

Models: learning from a Dataset

A Model is created from a Dataset.

Model_Manifest :: struct {
    project_id: string
    model_id: string
    display_name: string
    status: string
    dataset: Model_Dataset_Reference
    selected_model: string
}

The Model must identify:

Which Dataset was used?
Which model candidate was selected?
Is the model finalized?

A model that points to a missing Dataset cannot be trusted.

Model finalization

The Research index expects finalized Models to have:

status: finalized
selected_model: a non-empty model identifier

A final-test evaluation may also be stored:

models/model-001/evaluations/final-test.ramal-model-evaluation

That evaluation must identify:

• The Project.
• The Model.
• The final-test split.
• A complete status.

This separates “a model was trained” from “a model was evaluated for final use.”

Predictions: applying a Model

A Prediction stores the output of a finalized Model for a new configuration.

Prediction_Manifest :: struct {
    project_id: string
    prediction_id: string
    display_name: string
    status: string
    model: Prediction_Model_Reference
    dataset: Model_Dataset_Reference
}

A Prediction must point to both:

Model
Dataset

The Model and Dataset must agree. A Prediction cannot use a Model trained from one Dataset while claiming to use another.

Prediction feature values

The Prediction records the input features used by the Model:

Prediction_Feature_Value :: struct {
    feature_id: string
    unit: string
    value: f32
}

For example:

thickness = 25 cm
material  = concrete

The Research index checks that the Prediction uses the selected Model and its feature definitions.

Prediction verification

A Prediction is an estimate. Verification compares it with a real PHITS Result.

Prediction_Verification_Record :: struct {
    project_id: string
    prediction_id: string
    verification_id: string
    status: string
    prediction_path: string
    reportable_voxel_count: i32
    target_voxel_count: i32
    reportable_fraction: f32
}

Possible statuses include:

field-verified
does-not-meet-criteria

The verification record must identify:

• The Prediction.
• The Prediction path.
• The verification ID.
• The measured Result evidence.
• Reportable voxel coverage.

This closes the evidence loop:

Prediction
    ↓ compare with
PHITS Result
    ↓
Prediction Verification

The Research index

The Research_Index is RAMAL-EBX’s searchable view of research records.

Research_Index :: struct {
    entries: [dynamic]Research_Index_Entry
    integrity_ok: bool
}

Each entry records information such as:

kind
id
status
path
study_id
assessment_id
dataset_id
model_id
prediction_id
valid
reason

The index is rebuilt by scanning the Project directories.

It does not blindly list every file. It validates each record first.

Loading the Research index

At application startup, the index is initialized and loaded:

research_index_init(&app.research_index)
research_index_load(&app.project, &app.research_index)

The application also loads Studies and their catalogs:

study_catalog_init(&app.study_catalog)
study_catalog_load(&app.project, &app.study_catalog)

This happens in app_init_state, alongside the Case and Design catalogs.

If the Research index contains malformed entries, integrity_ok becomes false.

Scan order matters

The index scans records in dependency order:

Studies
    ↓
Datasets
    ↓
Models
    ↓
Predictions
    ↓
Prediction verifications

This is important because a Dataset can only be accepted after its Study is known to be valid.

A Model can only be accepted after its Dataset is valid.

A Prediction can only be accepted after both its Model and Dataset are valid.

research_index_scan_studies(project, &next)
research_index_scan_manifests(project, &next, .Dataset)
research_index_scan_manifests(project, &next, .Model)
research_index_scan_manifests(project, &next, .Prediction)

Invalid records are visible

When an entry fails validation, RAMAL-EBX records an invalid index entry:

research_index_append_invalid(
    index,
    .Dataset,
    id,
    path,
    "Dataset references a missing or invalid Study",
)

The entry contains a reason such as:

Dataset references a missing or invalid Study

This is useful for debugging. The application does not pretend the record does not exist; it marks it as unusable.

Safe deletion

Provenance also affects deletion.

RAMAL-EBX prevents deleting a Study when a Dataset references it:

Study
  ↑
Dataset

It prevents deleting a Dataset when a Model references it:

Dataset
  ↑
Model

It prevents deleting a Model when a Prediction references it.

This protects the chain of custody.

flowchart TD
    S[Study] --> D[Dataset]
    D --> M[Model]
    M --> P[Prediction]
    S -. cannot delete while referenced .-> D
    D -. cannot delete while referenced .-> M
    M -. cannot delete while referenced .-> P

If the index itself contains invalid entries, deletion is blocked until the integrity problem is resolved.

What happens when a source changes?

Consider this example:

1. A Result is verified.
2. An Assessment records its metrics.
3. The Result artifact is later replaced.
4. The Assessment is opened again.

During reload, RAMAL-EBX recomputes the metrics. If they differ from the stored metrics, the Assessment is rejected.

The same principle applies higher in the chain:

Changed Study
    ↓
Dataset provenance no longer matches
    ↓
Dataset is rejected

Changed Dataset
    ↓
Model dependency no longer matches
    ↓
Model is rejected

This is why provenance is more than storing filenames. It is active validation.

A complete beginner workflow

Here is the full workflow:

1. Create Cases from Designs.
2. Create Jobs and Attempts.
3. Publish verified Results.
4. Create a Study from compatible Cases.
5. Run the Study members.
6. Create an Assessment.
7. Recompute metrics from Results.
8. Build a Dataset from assessed members.
9. Train and finalize a Model.
10. Create a Prediction.
11. Run a new PHITS Case for comparison.
12. Create a Prediction Verification.

The GUI remains the supported entry point for these operations. Repository scripts support the application, but the research workflow is not complete until it can be performed through RAMAL-EBX.

Under the hood: ML task execution

Research commands run through ML_Task.

ML_Command :: enum {
    Build_Dataset,
    Train_Model,
    Evaluate_Final,
    Create_Prediction,
    Verify_Prediction,
}

The GUI starts one configured research command:

ml_task_start(
    &app,
    .Build_Dataset,
    spec_path,
    "",
    "",
)

RAMAL-EBX validates that the input files belong to the active Project before starting the process.

Research command outputs

Each command expects a specific output type:

Build Dataset       → .ramal-dataset
Train Model         → .ramal-model
Evaluate Final      → .ramal-model-evaluation
Create Prediction   → .ramal-prediction
Verify Prediction   → .ramal-prediction-verification

After a successful command, the application reloads the Research index:

if state.success && state.exit_code == 0 {
    task.phase = .Succeeded
    _ = research_index_load(&app.project, &app.research_index)
}

This makes newly created records visible to the application.

Research task safety

The ML task runner:

• Uses the Project as its working directory.
• Resolves input files inside the Project.
• Writes logs under research/.
• Rejects paths outside the Project.
• Reports missing Python dependencies.
• Leaves source records unchanged when stopped.

A stopped task reports:

ML task stopped; immutable source records were unchanged

This follows the same safety idea as immutable simulation records: derived work must not rewrite its source evidence unexpectedly.

Where this code lives

study.odin handles:

• Study definitions.
• Study members.
• Evaluation domains.
• Evaluation regions.
• Inclusion criteria.
• Study creation.
• Study catalogs.
• Study member Jobs.

study_run.odin handles:

• Sequential Study execution.
• Current member state.
• Success, failure, and stop counts.

assessment.odin handles:

• Assessment records.
• Result discovery.
• Evidence verification.
• Metric recomputation.
• Region means.
• Inclusion decisions.
• Assessment catalogs.

research_index.odin handles:

• Study, Dataset, Model, Prediction, and Verification indexing.
• Provenance checks.
• Dependency checks.
• Invalid-entry reporting.
• Safe deletion rules.

ml_task.odin handles:

• Dataset building.
• Model training.
• Final evaluation.
• Prediction creation.
• Prediction verification.
• Research task logs and process state.

app_state.odin connects these services to the application session.

A beginner’s mental model

Think of the pipeline as a laboratory filing system:

Study
    The research question and selected experiments

Assessment
    The inspection report for each experiment

Dataset
    A carefully labeled table built from accepted evidence

Model
    A trained tool linked to that table

Prediction
    A model estimate for a new configuration

Prediction Verification
    A comparison between the estimate and a real simulation

Every record carries labels pointing backward.

Every loader checks those labels.

Every derived record can be rejected if its source is missing, malformed, or inconsistent.

Conclusion

You learned that:

• A Study groups Cases and Jobs around a research question.
• Study domains and regions define where Results are evaluated.
• Inclusion criteria define what counts as reportable evidence.
• Assessments recompute metrics from verified Result artifacts.
• Stored assessment metrics are checked again when loaded.
• Datasets link samples back to Studies and Assessments.
• Models link back to Datasets.
• Predictions link back to Models and Datasets.
• Prediction Verifications compare predictions with PHITS evidence.
• The Research index loads records in dependency order.
• Invalid provenance makes a record unusable.
• Deletion is blocked when downstream records still depend on a source.
• ML tasks update the index after successful completion without changing immutable source records.

The Research Provenance and Evidence Pipeline turns isolated simulation files into a traceable scientific argument.

Next, we will see how analytical CSG geometry becomes visible triangles in the CSG Mesh and Rendering Pipeline.

---

Generated by AI Codebase Knowledge Builder