How much life is left in this part?

Nov 2026 — Essay — Bayesian Neural Networks

Imagine a model tells you that an aircraft part will last another 38,000 cycles. Do you believe it? What if it said “between 30,000 and 45,000”? What if it said “I have no idea, I’ve never seen a part like this”?

An ordinary neural network always answers with the same confidence. Whatever you ask, it returns a number, and that number says nothing about how much you should trust it. In many applications that doesn’t matter. In engineering, where a prediction decides when a machine is stopped, a bridge is inspected or a part is replaced, it is exactly what’s missing. And it gets expensive.

Unplanned downtime costs the world’s 500 largest companies around 11% of their revenue, about 1.4 trillion dollars a year. An hour of stopped production line ranges from roughly 36,000 dollars in consumer goods to over 2 million in automotive (Siemens, 2024). Getting ahead of a failure pays off: according to the US Department of Energy’s maintenance guide, predictive maintenance saves 8 to 12% compared with preventive maintenance and 30 to 40% compared with waiting for something to break (Sullivan et al., 2010).

But predicting is not enough: you have to know when to act. A metal part works under repeated loads. At some point a tiny crack appears and, cycle after cycle, it grows. While it is small nothing happens; when it reaches a critical length, the part breaks. You have measurements up to today and a very concrete question: when do I have to replace it?

Synthetic data of a growing crack. Switch between “a number” and “with uncertainty” and watch where the possible crossings of the critical length fall.

An ordinary model gives you a number: so many thousand cycles. It’s a good number, probably the most likely one. But it doesn’t tell you the only thing you need to decide: how much sooner it could happen. If you replace the part exactly at that number, in half of the cases it will already have broken. Studies that plan maintenance with probabilistic predictions confirm it: with aircraft engines, the optimal moment to replace falls close to the lower end of the remaining-life interval, not at its centre, and the cost is halved compared with replacing on a schedule (de Pater and Mitici, 2023).

Bayesian neural networks provide exactly that: instead of one answer, a distribution of answers. In this post I explain what they are, starting with the two pieces they are made of (Bayes and neural networks), why training them is hard, and how an algorithm I particularly like, ABC-SubSim, solves it. Then we go back to the part.

All the charts are computed in your browser while you read, with a fixed seed so you see the same thing I do.

Bayes: changing your mind with data

Bayes’ theorem comes from something almost obvious. The probability that two things, A and B, happen together can be counted in two ways: the probability of A times that of B given A, or that of B times that of A given B. Both counts give the same result, so you equate them, solve, and the theorem comes out. What it does is flip a conditional probability: if you know how often a cause produces an effect, it tells you how likely the cause is when you see the effect.

Written with θ for what you don’t know (a defect rate, the weights of a network) and “data” for what you have observed:

P(θ|data)=P(data|θ)·P(θ)P(data)

  • P(θ), the prior: what you believe before looking.
  • P(data | θ), the likelihood: if θ were true, how likely it would be to see exactly this data.
  • P(data), the evidence: a constant that makes the result add up to one.
  • P(θ | data), the posterior: what you believe after looking.

The likelihood is the key piece. It is the only term through which the data enter: you set the prior yourself, the evidence only normalises, and the likelihood decides how far your opinion moves. If two hypotheses explain what you have seen equally well, the data can’t tell them apart and you stay where you were. If one explains it twenty times better than the other, it gains weight twenty times faster. It is also the part that gets confused most: P(data | θ) is not P(θ | data), and the theorem exists precisely to get from one to the other. It will show up again in the Bayesian network, because it is the part that ABC replaces with a simpler question.

A software example: the spam filter

The best-known case of Bayes in software engineering is in your inbox. Bayesian spam filters became popular with “A Plan for Spam” (Paul Graham, 2002) and are still a component of tools like SpamAssassin. The idea, with round numbers:

  • Prior: 30% of the mail you receive is spam.
  • Likelihood: the word “free” appears in 40% of spam and in only 2% of normal mail.

An email arrives with “free”. Bayes says the probability that it is spam is 0.40 · 0.30 / (0.40 · 0.30 + 0.02 · 0.70) ≈ 90%. You have gone from 30% to 90% because of a single word, and what pushed it was the likelihood: “free” is twenty times more likely in spam than in normal mail. Notice also what it does not say: the fact that “free” appears in 40% of spam doesn’t mean an email with “free” is spam with 40% probability. That confusion between P(data | θ) and P(θ | data) is exactly what the theorem avoids.

A real filter does the same with hundreds of words at once, and every new word updates the probability again.

From a number to a curve

With spam, θ can only be “spam” or “not spam”. In engineering, θ is usually a number that can take any value. Let’s put it in a factory. You don’t know what percentage of parts comes out defective from a production line. You can start from what the supplier says, from a pessimistic suspicion or from no idea at all, and then inspect parts:

The line actually has an 8% defect rate. Change the number of inspected parts and what you believe at the start.

What matters is not the formula but what it produces: a curve, not a number. Its position tells you what is most likely; its width, how much you know. With twenty parts the curve is wide and honest (“somewhere between 4 and 27%”); with two hundred, narrow. That width is exactly what the crack prediction lacks.

A neural network, in brief

A neural network is a function with many parameters that are adjusted to imitate some data. It is made of very simple pieces, the neurons. Each one receives some numbers, multiplies them by some weights, adds them up, adds a bias and passes the result through a nonlinear function:

output=tanh(w1x1+w2x2+…+b)

If you stack layers of neurons, the output of one is the input of the next. With enough neurons, such a network can approximate almost any continuous function (Hornik et al., 1989). The one in this post is small on purpose: one input x, two layers of five neurons and one output ŷ.

The parameters

What the network learns are two kinds of numbers:

  • Weights (w): one per connection. They say how much one neuron influences the next and in which direction: a large weight amplifies the signal, one close to zero switches the connection off and a negative one inverts it. Here there are 35 (5 + 25 + 5).
  • Biases (b): one per neuron. They shift the point at which the neuron “switches on”, whatever the input. Here there are 11 (5 + 5 + 1).

In total, 46 parameters. There are other numbers that are not learned but chosen by you before training, the hyperparameters:

  • Architecture (how many layers and how many neurons): sets how much complexity the network can represent.
  • Activation function (here, the hyperbolic tangent): the one that bends the straight lines. Without it, stacking layers would be the same as having just one.
  • Learning rate (η): the size of each step when adjusting the parameters. If it is too small, the network barely advances; if it is too large, it overshoots.
  • Number of steps: how many times the adjustment is repeated.

How it learns

Training means looking for the 46 values that make the output resemble the data. It is done by repeating four steps:

  1. Forward pass: each data point enters on the left, goes through the layers and a prediction ŷ comes out.
  2. Error: ŷ is compared with the true value y. Here, with the mean squared error: the mean of (ŷ − y)² over all the data.
  3. Backpropagation: using the chain rule, it computes how much the error would change if each parameter were touched, its gradient (Rumelhart, Hinton and Williams, 1986).
  4. Update: each parameter takes a small step against its gradient, w←w−η·∂E∂w. This is gradient descent.

The figure trains a real network in your browser. It starts by doing three iterations slowly, marking each of the four steps; then it speeds up to 1,500 and, when it finishes, starts again from other random weights. The data are the same throughout the post: a noisy cosine, with a stretch where nothing was measured. We leave the “Bayesian” mode for the next section.

“Classic”: a network being trained with gradient descent. “Bayesian”: the 3,000 networks accepted by ABC-SubSim, one after another. The fill of each neuron is its activation; the thickness of each connection, the size of its weight (dashed if negative). Hover over a connection to see its value.

If you want more detail: the 3Blue1Brown series on neural networks is the best visual introduction; Michael Nielsen’s free book, Neural Networks and Deep Learning, explains backpropagation step by step; in TensorFlow Playground you can change the hyperparameters and see what happens; and Deep Learning, by Goodfellow, Bengio and Courville, is the complete reference.

A single network, too sure of itself

Look at what training returns: one value for each weight and therefore one curve. Let the figure train a few networks. Each one starts with different random weights and all of them end up fitting the points almost equally well. But in the gap with no data each one draws something different, and all of them do it with the same confidence. Gradient descent keeps one of them, almost by chance, and hands it to you as if it were the only one.

That unfounded confidence is the same as in the crack at the beginning, and it is fixed as in the factory: treating the weights as something we don’t know and applying Bayes to them (MacKay, 1992; Neal, 1996). That is what comes next. First, what changes when the network is Bayesian. Then, how it is computed, because that is where the difficulty lies: the posterior of 46 weights has no formula, and we are going to sample it with ABC-SubSim, an algorithm that uses no gradients. It generates random networks and, level by level, keeps the ones that fit.

Why Bayesian

Let’s put the two ideas together. If the weights are what we don’t know, they are our θ. We give them a prior (before seeing data, any weight between −1 and 1 is reasonable), and Bayes gives us a posterior: not a vector of weights, but a distribution over all the weight vectors compatible with the data.

That is what the “Bayesian” mode of the network figure shows. Instead of one network, you watch the 3,000 accepted by the algorithm go by, each with its 46 weights. The connections that change a lot from one network to another are weights about which the data say little; the ones that barely move are well determined. And below, their curves: nearly identical where there are points and fanning out in the gap.

To predict you no longer use one network: you use all the networks in that distribution and look at what they say. Where they agree, the prediction is reliable. Where they disagree, the disagreement is the uncertainty. Compare for yourself:

The data are a noisy cosine, and in the middle there is a stretch where nothing was measured. The Bayesian network doesn’t invent more confidence than it has: the band opens up right there.

That band is what the literature calls epistemic uncertainty: the kind that comes from not knowing, from not having enough data. It shrinks by measuring more. There is another, aleatoric uncertainty, which is the noise inherent to the phenomenon (two identical parts never fail on the same cycle) and which doesn’t go away however much you measure (Kendall and Gal, 2017). A good Bayesian network tells the two apart, and that distinction is gold in engineering: it tells you whether it is worth installing more sensors or whether you have already reached the limit of what is predictable.

The chart also shows a limit, and I prefer to show it rather than hide it: to the left of −3 and to the right of 3 there is no data either, and yet the band does not open up. All the networks agree on the same thing, and all of them are wrong. A Bayesian network only knows that it doesn’t know within the space that its architecture and its prior allow it to imagine. That is why, in production, uncertainty is validated just like accuracy.

How the posterior is computed

So far, the idea. Now, the problem. The posterior of a network has no formula: the evidence P(data) is an integral over all the weights at once, 46 dimensions in this network and millions in a real one. Nobody computes it exactly; it is approximated. The practical approximations fall into a few families:

  • MCMC sampling, above all Hamiltonian Monte Carlo. It builds a Markov chain that, if left to run long enough, visits the weights in proportion to their posterior probability. HMC uses gradients to make long, informed jumps. It is the gold standard in quality (Neal, 1996), but it is expensive and hard to tune on large networks.
  • Variational inference, such as Bayes by Backprop (Blundell et al., 2015). It picks a simple family, usually an independent Gaussian per weight, and adjusts its parameters with gradients so that it resembles the posterior as closely as possible. It scales well, but the family imposes its shape and tends to underestimate the uncertainty.
  • MC Dropout (Gal and Ghahramani, 2016). It keeps dropout (switching neurons off at random) on at prediction time too and treats each pass as a sample. It is almost free if the network already uses dropout, but the quality of the approximation depends a lot on the architecture.
  • Laplace approximation (MacKay, 1992; Daxberger et al., 2021). It trains the network normally and places a Gaussian around the optimum, wider or narrower depending on the curvature of the error. It is applied to an already trained network, but it only sees the valley where it landed.
  • Ensembles (Lakshminarayanan et al., 2017). They train several networks from different starting points and query them all at once: it is what the training figure did when it started again from other weights, but on purpose. It is not strictly Bayesian, but in practice it works very well.
  • ABC-SubSim (Chiachío et al., 2014; applied to networks by Fernández et al., 2022). Level-by-level sampling, with no gradients and no explicit likelihood. It is the one I use in this post.
MethodGradientsShape of the posteriorLarge networks
HMCYesFree (samples)Very expensive
VariationalYesImposed (usually Gaussian)Yes
MC DropoutYes, to trainImplicit in the dropoutYes
LaplaceYes, and curvatureLocal GaussianYes, with approximations
EnsemblesYesA few networksYes, costs N trainings
ABC-SubSimNoFree (samples)Small and medium

A table like this simplifies a lot, but it shows the pattern: almost all the methods rely on gradients, the same tool as classic training. The column I care about is the second one. ABC-SubSim is the only one on the list that needs no derivatives, and it comes from an unexpected place: reliability engineering, the same discipline that asks when a part breaks.

ABC-SubSim, step by step

ABC: swapping the likelihood for a question

In the Bayes section I said the likelihood is the key piece, because it is where the data come in. Approximate Bayesian Computation (ABC) replaces it with a yes-or-no question: does what this network predicts resemble the data, with an error smaller than a threshold ε? If the answer is yes, the network is accepted; if not, it is discarded. The smaller ε is, the closer the result is to the exact posterior.

The direct version is very simple: generate networks with weights drawn from the prior and keep the ones that meet the criterion. It needs no gradients and no formula for the likelihood, just the ability to compute the error. Its problem is the acceptance rate. With a demanding ε, hitting the target blindly is a rare event: in this experiment, none of 3,000 random networks comes close to the final threshold. You would have to generate hundreds of millions to accept one.

Subset Simulation: a ladder towards the rare

Estimating very small probabilities is a classic problem in structural reliability: the probability that a bridge collapses is, by design, tiny. Subset Simulation (Au and Beck, 2001) writes the rare event F as the last of a chain of nested events, F1⊃F2⊃…⊃F, and factorises:

P(F)=P(F1)·P(F2|F1)·…·P(F|Fm−1)

Instead of asking at once for something with a one-in-a-billion probability, you ask for several things in a row with a 20% probability each. None of them is rare, so none requires blind simulation: each level is filled starting from the samples of the previous level that already qualify.

ABC-SubSim (Chiachío et al., 2014) applies that ladder to ABC. The nested events are the regions of weights that meet ever smaller error thresholds, D1⊃D2⊃…, and each rung lowers ε. With two parameters it is very clear:

A toy problem with two parameters. Each ellipse is an acceptance region Dⱼ; each level keeps roughly 20% of the previous one and spreads it out again inside.

The algorithm

  1. Generate N networks with weights drawn from the prior and compute the error of each one.
  2. Sort them by error and keep the best fraction P0 as seeds. The level’s threshold ε is the error of the first one left out.
  3. From each seed, run a short chain: try small variations of its weights. If the variation stays below ε, move on; if not, stay where you were.
  4. With the seeds and their chains you again have N networks, all below ε. Repeat from step 2, lowering the threshold and tuning the size of the variations.

Step 3 doesn’t look for the best network, but for many networks that meet the threshold, spread over the whole acceptable region. That is what turns the method into a sampling of the posterior and not an optimisation in disguise. The variations follow the modified Metropolis of Au and Beck: each weight moves or not according to its prior, and the complete proposal is only accepted if its error does not exceed ε. That way the chain respects the prior without needing the likelihood.

The parameters

Just as the network had its hyperparameters, the algorithm has its own. These are the ones behind all the charts in the post:

  • Error function: mean squared error over the 32 points. It is the only thing the method needs from the data, and it takes the place of the likelihood. It can be any measure: a custom metric, or the output of a finite element simulator.
  • Prior: a normal truncated to [−1, 1] for each of the 46 parameters (standard deviation ≈ 0.54). It sets where to search: a weight outside that interval is never proposed. It is also what shapes the uncertainty where there is no data.
  • N = 3,000 networks per level. This is the population size, and therefore the size of the final posterior sample. With more networks, the posterior is better described and the tails are easier to see, but the cost grows proportionally.
  • P0 = 0.2: the fraction that survives each level. With N = 3,000 that gives 600 seeds, and each one runs a chain of 1/P0 − 1 = 4 steps to get back to 3,000. With a small P0 you descend faster, but each level starts from fewer seeds and the networks look more alike. The usual range is between 0.1 and 0.3.
  • ε, the threshold: it is not chosen. It is computed at each level as the error that leaves out 80% of the population. That is why the ladder adapts to the problem by itself.
  • Levels = 12. When to stop. Each level keeps roughly 20% of the previous one, so twelve levels reach a region that occupies 0.212, about four billionths of the prior. Direct ABC would need around 240 million networks for each one accepted.
  • σ, the size of the variations: from 0.3 to 0.05. At the start the acceptable region is wide and it pays to take big steps to explore it; at the end it is narrow and a big step almost always falls outside. If σ is too large, the chain almost never accepts; if it is too small, it barely moves. The acceptance rate of each level tells you whether it is well chosen.
  • Seed = 7. It fixes the random numbers so that you see exactly the same charts I do.

With the 46-parameter network the regions can’t be drawn, but what happens at each level can. At the top, a sample of the population of networks; below, the error of all 3,000 and the threshold set by each level. As you advance, the histogram shifts to the left and narrows, and the curves at the top go from random scribbles to following the data. Notice that in the gap they don’t agree: that disagreement is the band of the Bayesian network.

Press “Play”. At the top, 40 of the 3,000 networks of each level. Below, the histogram of the error and the thresholds εⱼ, on a logarithmic scale: each tick is ten times smaller.

Three properties that justify using it in engineering:

  1. No gradients and no likelihood. All you need is an error function. If the network feeds a simulator or the error is a metric that can’t be differentiated, it doesn’t matter.
  2. Free-form posterior. The final population can be asymmetric, have several peaks or heavy tails. Nothing forces it to be a Gaussian bell, which is exactly what variational inference and Laplace do.
  3. Evidence for free. Since each level keeps a known fraction of the previous one, their product estimates the evidence P(data) for the final threshold. That makes it possible to compare architectures or priors with a Bayesian criterion.

Back to the part: deciding under uncertainty

Let’s go back to the opening question, now as a decision. You have to set a week for the maintenance stop. There is a company rule: the risk that the part fails before the stop cannot exceed 5%. A deterministic network gives you its prediction (week 38) and nothing else. A Bayesian one gives you the cumulative probability of failure week by week:

Illustrative data. Move the stop and compare the risk estimated by each network.

With the deterministic network, the risk is zero until the eve of its prediction and a hundred after. The 5% rule can’t even be applied: either you stop “just in case” with a margin somebody makes up, or you push on to week 37. With the Bayesian one the rule applies itself, and it proposes stopping earlier. Not because it is pessimistic, but because it knows that week 38 is a median and not a guarantee.

That is the real value of uncertainty in industry: it turns a prediction into a defensible decision. And the pattern repeats across many sectors:

  • Predictive maintenance of turbines, bearings or compressors: planning stops and spares according to risk, not according to the mean prediction.
  • Material fatigue: expensive, highly scattered tests, where the bands matter as much as the mean. There is work applying precisely Bayesian networks trained with ABC-SubSim to composite materials (Fernández et al., 2022).
  • Energy: sizing reserves or bidding in the market with the plausible range of demand and renewable production, not with a single value.
  • Quality and inspection: having the vision system abstain and pass the part to a person when it is in doubt.
  • Structures: telling an alarm caused by real damage from one caused by a temperature change, knowing how much to trust each.

A good idea with a price

I find the idea very good: it turns the question “which is the network?” into “which networks are compatible with what I have seen?”, and it does so with a tool that only needs to measure an error. The problem is what it costs.

  • Many more evaluations. Level 0 evaluates 3,000 networks and each of the following 12, 2,400 more: about 32,000 networks, each over the 32 points, around a million forward passes. The gradient descent of the training figure does 1,500 steps over those same 32 points: between ten and twenty times less work, counting the backward pass. Here, with 46 parameters, that is under a second in your browser. With a real network, each pass costs millions of times more.
  • It scales badly with the number of parameters. Random variations work well in 46 dimensions. In a million, almost any step that isn’t tiny falls outside the acceptable region, and with tiny steps the chain barely moves. That is why the methods that scale use gradients: they know where to go.
  • Memory and prediction. The result is not one network, but 3,000. With 46 parameters that is 138,000 numbers; with a million parameters per network it would be 3 billion, about 12 GB. And each prediction requires querying all of them, or a representative sample.
  • The prior matters. It has to be chosen carefully, because it decides which networks can ever be proposed and how the band opens up where there is no data.

It is not a problem exclusive to ABC-SubSim. Sampling the posterior of a large network well is expensive with any method: to study what the posterior of modern networks is really like with HMC, Izmailov et al. (2021) needed hundreds of TPUs. The cheap alternatives, such as variational inference or MC Dropout, are cheap because they approximate more.

In ABC-SubSim’s favour, it is very easy to parallelise: the chains of each level are independent of one another. That is why it fits so well the kind of problem in this post, which is the usual one in engineering: small or medium models, little data that is expensive to obtain, and decisions where being wrong is costly. A crack-growth curve, a fatigue test or a vibration signal don’t need millions of parameters, and there what matters is being able to trust the uncertainty. Which, by the way, also has to be checked: 90% of new cases should fall inside the 90% band.

When what is at stake is stopping a line, inspecting a bridge or replacing a part on an aircraft, the question is never just “what is going to happen?”. It is “what is going to happen, and how much are you betting that you’re right?”. A Bayesian network answers both.

References

  1. Au, S. K. and Beck, J. L. (2001). Estimation of small failure probabilities in high dimensions by subset simulation. Probabilistic Engineering Mechanics, 16(4), 263–277.
  2. Basora, L., Viens, A., Arias Chao, M. and Olive, X. (2023). A benchmark on uncertainty quantification for deep learning prognostics.
  3. Blundell, C., Cornebise, J., Kavukcuoglu, K. and Wierstra, D. (2015). Weight uncertainty in neural networks. ICML.
  4. Chiachío, M., Beck, J. L., Chiachío, J. and Rus, G. (2014). Approximate Bayesian computation by subset simulation. SIAM Journal on Scientific Computing, 36(3).
  5. Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M. and Hennig, P. (2021). Laplace redux: effortless Bayesian deep learning. NeurIPS.
  6. de Pater, I. and Mitici, M. (2023). Predictive maintenance planning using renewal reward processes and probabilistic RUL prognostics.
  7. Fernández, J., Chiachío, M., Chiachío, J., Muñoz, R. and Herrera, F. (2022). Uncertainty quantification in neural networks by approximate Bayesian computation: application to fatigue in composite materials. Engineering Applications of Artificial Intelligence, 107.
  8. Gal, Y. and Ghahramani, Z. (2016). Dropout as a Bayesian approximation: representing model uncertainty in deep learning. ICML.
  9. Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning. MIT Press.
  10. Hornik, K., Stinchcombe, M. and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366.
  11. Izmailov, P., Vikram, S., Hoffman, M. D. and Wilson, A. G. (2021). What are Bayesian neural network posteriors really like? ICML.
  12. Kendall, A. and Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? NeurIPS.
  13. Lakshminarayanan, B., Pritzel, A. and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS.
  14. MacKay, D. J. C. (1992). A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3), 448–472.
  15. Neal, R. M. (1996). Bayesian Learning for Neural Networks. Springer.
  16. Nielsen, M. (2015). Neural Networks and Deep Learning. Determination Press.
  17. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
  18. Siemens (2024). The True Cost of Downtime 2024.
  19. Sullivan, G. P. et al. (2010). Operations & Maintenance Best Practices: A Guide to Achieving Operational Efficiency, Release 3.0. U.S. Department of Energy.