Engineering Data Science Analysis Protocol

Workflow map

From business question to sustained decision

1. Purpose and Scope

This protocol defines the minimum analytical steps required before engineering data is used to support a process change, quality disposition, operating recommendation, technical risk assessment, predictive model, optimization study, or management decision.

It is suitable for analyses involving:

2. Governing Principles

3. End-to-End Workflow

Phase 0 — Analysis Intake

Before touching the data, record the request in a structured intake form.

• Project title and analysis owner
• Decision-maker and affected stakeholders
• Operational or business decision to be supported
• Required completion date
• Consequence of a wrong conclusion
• Available data sources and known limitations
• Required output format: technical report, presentation, dashboard, model, or recommendation

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Create a complete analysis intake from the information I provide. Identify missing information before proceeding.
2. List every stakeholder, decision, risk, and required deliverable for this project.
3. Rewrite my problem statement into a precise engineering decision statement.
4. Identify ambiguities in my request that could invalidate the analysis later.
5. Generate a project charter using engineering best practices.

Phase 1 — Problem Definition

1.1 Convert the request into an analytical question

A strong problem statement identifies the response, predictors, population, comparison, decision threshold, and operating context.

Weak:
Determine whether temperature affects Charpy energy.

Strong:
Estimate the effect of exit temperature on transverse base-metal Charpy energy
after controlling for grade, OD, wall thickness, rolling practice, slab thickness,
chemistry, coiling temperature, and test temperature. Quantify the probability
that a proposed X65 product will meet the specified average and minimum requirements.

1.2 Define the analysis framework

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Convert this business problem into statistically testable hypotheses.
2. Identify the response variable, predictors, confounders, and unit of analysis.
3. Challenge my assumptions and propose alternative explanations.
4. Determine the most appropriate analytical objective and success criteria.
5. Identify decisions that depend on this analysis and their risks.

Phase 2 — Data Acquisition and Lineage

2.1 Establish the data grain

Every table must have a clearly defined grain. Combining tables of different grain without aggregation rules can create duplicate observations and false statistical confidence.

• Document the row definition for every source table
• Identify primary and foreign keys
• Confirm one-to-one, one-to-many, or many-to-many joins
• Test whether joins duplicate the response variable
• Record all filters and exclusions
• Preserve a raw, unchanged extract

2.2 Data lineage record

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Design the ideal dataset schema and required joins for this analysis.
2. Review my SQL query for duplicate risks and grain mismatches.
3. Identify missing source tables or variables that should be collected.
4. Produce a complete data lineage document.
5. Audit my joins for one-to-many and many-to-many problems.

Phase 3 — Data Quality Gate

No model should proceed until the dataset is assessed against the following dimensions.

3.1 Missing data analysis

Report missingness by variable, time period, product, grade, asset, and operational subgroup. Determine whether data are plausibly missing completely at random, missing at random, or missing not at random.

3.2 Outlier handling

Do not remove an observation only because it is statistically unusual. Classify each outlier as:

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Perform a complete engineering data-quality audit before modeling.
2. Identify missing values, duplicates, impossible values, and inconsistent units.
3. Look for data leakage and hidden sampling bias.
4. Recommend treatment for every outlier with justification.
5. Determine whether this dataset is fit for purpose.

Phase 4 — Exploratory Data Analysis

Exploration is used to understand the data structure. It is not the final basis for causal conclusions.

Minimum exploratory outputs

• Response distribution and subgroup distributions
• Time-order plot
• Scatterplots for continuous predictors
• Boxplots or interval plots for categorical predictors
• Correlation matrix for numeric variables
• Counts per subgroup and predictor combination
• Missingness map
• Outlier and leverage screening
• Process sequence or campaign effects
• Potential nonlinear patterns and interactions

Confounding and Simpson’s paradox checks

For each apparent relationship, stratify or condition on major process variables. A pooled trend may disappear or reverse when examined within grade, product, rolling practice, slab thickness, line, or time period.

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Perform exploratory analysis without drawing conclusions.
2. Identify hidden clusters, subgroup effects, and nonlinear relationships.
3. Test for Simpson's paradox and confounding.
4. Recommend the most informative visualizations.
5. Summarize insights that require confirmatory testing.

Phase 5 — Feature Engineering

Derived variables should reflect process knowledge rather than arbitrary mathematical manipulation.

Acceptable feature categories

• Unit conversions and standardized units
• Process deltas, such as exit minus coiling temperature
• Rates, ratios, residence times, and specific energy
• Chemistry equivalents and metallurgical indices
• Lagged or rolling process variables
• Product geometry terms
• Interactions supported by engineering mechanisms
• Grouped categorical factors for equipment, practice, or campaign

Feature rules

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Recommend engineered features based on process knowledge.
2. Identify useful interaction terms and polynomial effects.
3. Detect redundant predictors and multicollinearity risks.
4. Explain every engineered feature in engineering language.
5. Prioritize features by expected predictive value.

Phase 6 — Model Strategy and Comparison

The model family must match the response type, data structure, and decision objective.

6.1 Baseline model

Always fit a simple baseline model. Complex methods must demonstrate a meaningful improvement over the baseline.

6.2 Model comparison criteria

• Prediction error on unseen data
• Adjusted R², AIC, BIC, or likelihood-based criteria where appropriate
• Calibration
• Residual behavior
• Stability across subgroups and time periods
• Interpretability
• Operational deployability
• Consistency with engineering knowledge

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Compare all appropriate statistical and machine-learning models.
2. Recommend the simplest defensible model and explain why.
3. Rank candidate models by accuracy, interpretability, and deployment risk.
4. Identify assumptions for each model family.
5. Explain when a mixed-effects model is preferable.

Phase 7 — Validation and Stress Testing

7.1 Select the correct validation design

7.2 Regression diagnostics

• Residuals versus fitted values
• Normal probability plot where inference depends on normality
• Constant variance
• Independence and time-order behavior
• Influential observations and Cook’s distance
• Leverage
• Variance inflation factors
• Specification errors and omitted nonlinear terms

7.3 Sensitivity analysis

• Fit with and without influential observations
• Compare alternative subgroup definitions
• Test alternate missing-data treatments
• Test alternate model forms
• Test plausible measurement error
• Evaluate prediction under nearby operating conditions
• Assess whether conclusions depend on a small number of groups

7.4 Extrapolation control

For every prediction, compare requested predictor values with the historical data range and combinations. A value may be within each individual variable’s range but still represent an unseen combination.

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Design the best validation strategy for this dataset.
2. Perform a complete regression diagnostic review.
3. Stress-test the model against alternative assumptions.
4. Evaluate extrapolation risk for requested predictions.
5. Recommend sensitivity analyses before approval.

Phase 8 — Engineering Interpretation

Translate statistical output into process language.

Required interpretation questions

• What physical mechanism could explain the relationship?
• Is the direction of effect credible?
• Is the magnitude practically important?
• Does the effect hold within relevant product and process groups?
• Could the variable be acting as a proxy for another process condition?
• Is the effect controllable in operation?
• Would changing this variable create a trade-off elsewhere?

Coefficient interpretation format

Statistical statement:
Holding the other model variables constant, a 10°C increase in exit temperature
is associated with an estimated 6 J reduction in average Charpy energy.

Engineering statement:
Within the historical operating range represented by the model, higher exit
temperature is associated with lower toughness after accounting for the listed
process and chemistry variables. This relationship should not be interpreted
as causal unless supported by a controlled trial or stronger process evidence.

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Translate every statistical result into engineering language.
2. Explain whether the observed effects are physically plausible.
3. Identify operational actions supported by the model.
4. Separate correlation from likely causation.
5. Explain coefficient magnitudes in practical process terms.

Phase 9 — Decision and Risk Analysis

Convert the model into a decision statement, not merely a statistical conclusion.

Minimum decision outputs

• Point prediction or estimated effect
• Confidence interval
• Prediction interval where an individual future result matters
• Probability of meeting the specification or target
• Probability of an adverse outcome
• Sensitivity to uncertain inputs
• Extrapolation classification
• Operational recommendation
• Required safeguards or verification plan

Risk classification

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Convert the model into an executive decision recommendation.
2. Estimate probability of meeting specification.
3. Quantify operational and business risks.
4. Recommend implementation safeguards.
5. Summarize uncertainty for management.

Phase 10 — Documentation and Reproducibility

Required analysis package

• Approved problem statement
• Data dictionary
• Source query or extraction logic
• Raw and cleaned dataset references
• Data quality report
• Exploratory visuals
• Model specification and settings
• Diagnostics and validation results
• Sensitivity analysis
• Final engineering interpretation
• Decision recommendation
• Limitations and unresolved risks
• Reproducible code or software instructions
• Version and review history

Software-equivalent requirements

Where practical, provide:

AI Prompt Library

Use or adapt the following prompts to obtain high-quality, structured assistance from AI during this phase.

1. Create a fully reproducible analysis package.
2. Generate Python, Minitab, Excel, and SQL implementation guidance.
3. Produce executive and technical summaries.
4. List every limitation and assumption.
5. Create a peer-review-ready report.

4. Final Peer-Review Checklist

Before release, perform an adversarial review using the following questions.

• Could the conclusion be caused by confounding?
• Could Simpson’s paradox be present?
• Was the data grain handled correctly?
• Were repeated observations treated as independent?
• Could data leakage inflate performance?
• Does the validation method match the data structure?
• Are influential points driving the result?
• Are key predictor combinations absent from the historical data?
• Is the model extrapolating?
• Are statistical significance and practical significance being confused?
• Are assumptions supported by diagnostics?
• Does the physical mechanism make sense?
• Are uncertainty and limitations clearly communicated?
• Can another analyst reproduce the result?
• Would the recommendation change under a reasonable alternative model?

5. Master ChatGPT Prompt

Use the following prompt at the start of a new engineering analysis.

You are acting as my senior engineering data scientist and independent technical reviewer.

Your role is to help me produce a statistically valid, physically credible, reproducible,
and decision-oriented analysis.

Follow the Engineering Data Science Analysis Protocol below.

1. Begin by restating the business decision, analytical question, response variable,
   unit of analysis, population, success criterion, and consequence of a wrong conclusion.
2. Create a data lineage and data-quality plan before modeling.
3. Audit missing values, duplicate business keys, incorrect joins, impossible values,
   inconsistent units, outliers, measurement-system issues, time effects, subgroup
   imbalance, data leakage, confounding, sampling bias, and extrapolation risk.
4. Separate exploratory analysis from confirmatory analysis.
5. Test alternative explanations including grade, product, geometry, practice, equipment,
   campaign, chemistry, time, and measurement effects.
6. Recommend the simplest valid baseline model and compare stronger candidates where appropriate.
7. Select validation that respects grouped, repeated, or time-ordered data.
8. Check assumptions, residuals, influence, multicollinearity, calibration, stability, and sensitivity.
9. Do not interpret correlation as causation.
10. Translate findings into engineering and operational language.
11. Report point estimates, intervals, probability of meeting specification, practical
    significance, extrapolation status, risk classification, limitations, and action.
12. Attempt to disprove the preferred conclusion before accepting it.
13. Provide reproducible Python, Minitab, Excel, and SQL implementation where applicable.
14. Stop and report any condition that makes the analysis unreliable.

Project title:
Business decision:
Primary question:
Response variable:
Potential predictors:
Unit of analysis:
Population:
Specification or target:
Known process knowledge:
Known confounders:
Data sources:
Required deliverables:
Decision deadline:

6. Reusable Templates

6.1 Project charter

Project title:
Owner:
Decision-maker:
Business decision:
Primary analytical question:
Response variable:
Predictor variables:
Unit of analysis:
Population:
Date range:
Success criterion:
Risk of false positive:
Risk of false negative:
Known confounders:
Expected deliverables:
Approval authority:

6.2 Data-quality summary

Dataset:
Source:
Row count:
Date range:
Business key:
Duplicate count:
Missingness:
Unit issues:
Outlier count:
Join risks:
Measurement risks:
Subgroup imbalance:
Extrapolation concerns:
Disposition:
- Fit for purpose
- Fit with limitations
- Not fit for purpose

6.3 Model comparison table

6.4 Final recommendation format

Decision:
Recommended action:
Estimated result:
Uncertainty:
Probability of meeting target:
Historical support:
Extrapolation status:
Primary risk:
Key assumptions:
Required controls:
Verification plan:
Conditions that would change the recommendation:

7. Standard Deliverables