Running and Re‑running Jobs for Reproducible Analysis

Core Concepts

• Projects hold your data (raw, processed, work, metadata).
• Jobs are executions of analysis templates (e.g. unpack, DIALS).
• Reproducibility means:
  • Every job keeps its inputs, parameters, logs, template version, and outputs.
  • You can re‑run the same job with modified parameters.
  • All iterations are grouped in a job history.

You do not need to manage hardware directly; compute resources are handled automatically.

Step 1 : Run a Job

1. Upload data

1. Go to your Project space.
2. Navigate to the appropriate folder (e.g. raw/).
3. Upload your data.
   • Best practice: upload a single ZIP file if you have many small files. This is faster and more reliable.

2. Start an analysis job

1. Go to the Analysis tab.
2. Select a public template (e.g. Unpack, DIALS macromolecule tutorial).
3. Click Run job.
4. Select the input data:
   • For unpacking: select the ZIP file.
   • For analysis (e.g. DIALS): select the folder containing the unpacked data, not individual files.
5. Review optional settings (machine type, sponsor, template version).
6. Start the job.

3. Monitor job status

After starting a job, it appears in the Jobs table with one of the following states:

• Pending = compute resources are being allocated and the environment is setup.
• Running = the job is executing.
• Completed = the job finished successfully.
• Failed = the job stopped due to an error.

Pending can take some time depending on resource availability.

Step 2 : Inspect Job Details

Click the eye (details) icon in the Jobs table to open the job details page.

You can inspect:

• Inputs – data paths and parameters used.
• Logs – live and final execution logs (essential for debugging).
• Outputs – files written during execution (may appear while the job is still running).
• Resource usage – CPU and runtime information.

👉 If a job fails, always check the Logs tab first.

Step 3 – Re‑run a Job with New Parameters

Re‑running is the core reproducibility feature.

How to re‑run

1. Open the job details page.
2. Click Rerun.
3. Adjust input parameters (e.g. space group, symmetry options, flags).
4. Start the new run.

Important notes:

• A re‑run creates a new job, it does not overwrite the original.
• Multiple re‑runs can execute in parallel.
• You can iterate as many times as needed.

Step 4 – View Job History (Iteration Tracking)

To compare iterations:

1. Open a job’s details page.
2. Click History (top‑right actions).

This shows a filtered view containing:

• The original job
• All its re‑runs
• Each set of parameters and outcomes

This makes it easy to:

• Compare different parameter choices
• See which iteration succeeded or failed
• Understand how results evolved

Common Workflow Pattern

1. Run a job with minimal or default parameters.
2. Inspect logs and outputs.
3. Identify missing or incorrect parameters.
4. Re‑run the job with corrected inputs.
5. Repeat until results are satisfactory.

This mirrors real scientific analysis and preserves full traceability.

Practical Tips

• ZIP before upload if you have many small files.
• Select folders, not files, when a tool expects a dataset.
• If a job fails:
  • Check logs first.
  • Look for parameter type errors (e.g. boolean vs numeric).
• Pending jobs are normal; resource allocation can take time.
• Re‑runs may start faster than original jobs.

Mental Model to Remember

• Data lives in projects.
• Software lives in job templates.
• Experiments happen as jobs acting on data.

Reproducibility is achieved through job history and preserved parameters.