All DevOps case studies / Case study 04 / 05
Pivelo Software
Self-healing CI/CD pipeline
A pipeline that understands why it failed, and fixes itself when it can
In short
When a build fails, an agent reads the thousands of log lines, finds the real cause, retries transient failures (network, timeouts) on its own and sends the developer a fix suggestion when the problem is in the code.
- Client
- Pivelo Software
- Sector
- Software publisher
- Size
- 120 developers, 900 pipelines a day
- −60%time lost on failed builds
- 70%of transient failures retried with no one involved
- 30 sto a diagnosis, instead of 20 minutes
The context
At Pivelo, roughly one pipeline in six failed, often for reasons unrelated to the code: registry down, flaky test, network timeout. Developers lost hours digging through ten-thousand-line logs.
The challenge
- 01
Logs are huge, and the useful line is buried in the middle.
- 02
Retrying blindly hides real bugs.
- 03
Developers need the information where they work.
- 04
The same failures come back, and no one remembers them.
Our approach
A FastAPI service receives GitLab CI failure events. A language model reads the logs, isolates the error trace and compares it with past failures stored in a vector database. If it’s a known transient error, it retries the job. If the problem is in the code, it sends the developer concerned the explanation and a suggested fix in Slack.
The solution, step by step
- 01
A pipeline fails
10,000 log lines, and one stage in red.
- 02
The cause isolated
The agent finds the two lines that matter amid the noise.
- 03
Compared with the past
A network timeout already seen 214 times: it’s transient.
- 04
Retried automatically
The job restarts and turns green, bothering no one.
- 05
Or fixed with the developer
If it’s a bug, the suggested fix lands in Slack.
Results
- −60%time lost on build failures
- 70%of transient failures solved alone
- 30 saverage diagnosis
- −40%“CI is broken” tickets
What we deliver
- Failure event service (FastAPI)
- Log analysis by language model
- Vector database of past failures
- Guarded automatic retries
- Fix suggestions in Slack
Technologies
- GitLab CI
- GitHub Actions
- Python
- FastAPI
- Claude API
- Qdrant
Timeline
6 to 8 weeks
Team
1 DevOps engineer, 1 AI engineer
Illustrative cases: clients and figures are fictional, representative of the solutions we deliver.