ENGINEERING TRANSPARENCY
What Has Been Tested. What Has Not. What Comes Next.
Most vendors bury their assumptions in fine print. This page states them in the first paragraph. Engineers who clone the repository and run the harness will find exactly what is documented here — no more, no less.
In systems engineering, the difference between a simulation result and a hardware-measured result is not a footnote. It is the entire question. This page defines two evidence classes, states which results belong to each, and lays out the next validation steps as a concrete roadmap that outside engineers can run and contribute to.
TWO EVIDENCE CLASSES
Class S and Class P
Every benchmark result on this site belongs to one of two classes. The class is not a quality judgment. It is a statement of what was measured and how.
Class S
Sealed Simulation
Results produced by the BenchmarkEngine running controlled twin-arm experiments on the same physical hardware, with both arms executing identical workloads under identical conditions. The governing engine is compiled C++ running on real silicon. The workload is real. The hardware is real. The measurement is real. What is controlled is the experimental setup: both arms run on the same machine, in the same thermal state, with the same input distribution. This eliminates hardware variance as a confound. The results are sealed with a cryptographic fingerprint and published to a public GitHub repository. Any engineer can clone the repository, run the harness, and verify byte-for-byte parity with the published fingerprint.
WHAT IS CLASS S
All current published benchmark results: +20.0% whole-system Omni index GAIN, +22.2% Kubernetes scheduling efficiency, −1.3% GPU energy per request (GAIN), +1.2% through-cloud bill (inside noise), −11.0% cloud 99th-pct response time (beyond noise). 17 sealed twin pairs. 97 paired repetitions. Monte Carlo validated. Sealed Nov 12, 2026.
WHAT IS NOT CLASS S
Live production trace replay. Multi-tenant cluster deployment. Cross-vendor hardware comparison. Long-duration soak tests on production infrastructure. These are Class P targets.
Class P
Physical Production
Results produced by deploying the governing engine into a live production environment and measuring its effect on real workloads over real time. Class P results require access to production infrastructure, real traffic, and a controlled rollout methodology. They are the gold standard for enterprise adoption. They are also the most expensive and time-consuming evidence to produce. Class P validation is the next chapter — not because the Class S results are insufficient, but because Class P results answer a different question: not 'does the engine work under controlled conditions' but 'does the engine work on your infrastructure, with your workload, at your scale.'
WHAT IS CLASS P
The next validation steps listed below. These are not speculative — they are the natural extension of the Class S methodology to live production environments.
WHAT IS NOT YET CLASS P
Not yet available. This is stated explicitly because engineers who find this caveat in fine print lose trust. Engineers who find it in the first paragraph gain it.
THE VERIFICATION ENTRY POINT
python verify.py
One command. Three checks. Any engineer can run it on day one.
$ git clone https://github.com/The-Omni-Compass-LLC/The-Omni-Compass
$ cd The-Omni-Compass
$ python verify.py
✓ C++ governor compiled — fingerprint matches SEAL.json
✓ Python parity verified — 10,000 decisions, 0 divergences
✓ Soak test passed — 100M decisions, Lyapunov monotone, no conservation violations
01
Compile the C++ Governor
The verify script compiles the omni_controller C++ source from the public repository using the same build flags as the sealed benchmark run. The compiled binary is hashed. The hash is compared against the published fingerprint in SEAL.json. A mismatch fails the verification immediately with a diff of the diverging bytes.
02
Verify Python Parity
The Python reference implementation in omni_controller/controller.py is a line-for-line translation of the C++ governor. The verify script runs both implementations against the same 10,000-decision input sequence and compares outputs decision-by-decision. Any divergence — including floating-point rounding differences above the tolerance threshold — fails the verification with the first diverging decision index.
03
100-Million-Decision Soak Test
The verify script runs the compiled C++ governor through a 100-million-decision soak test using the same Monte Carlo input distribution as the sealed benchmark. The soak test verifies three properties: the Lyapunov function decreases on every admitted decision, no conservation law is violated above the tolerance threshold, and the throughput gain over the baseline policy falls within the sealed result confidence interval. A failure on any of the three properties fails the verification with the specific decision index and property that failed.
HARDWARE-IN-THE-LOOP ROADMAP
The Next Evidence Steps
These are not aspirational. They are the natural extension of the Class S methodology to live production environments. Each step is a concrete test that outside engineers can run and contribute to.
Phase 1
Production Trace Replay
Replay published production traces from open-source Kubernetes clusters (the Alibaba Cloud trace dataset, the Google cluster trace) through the governing engine and compare scheduling decisions against the original scheduler's decisions. This is a Class S test on real-world input distributions rather than synthetic Monte Carlo inputs. It answers the question: does the engine make better decisions than the original scheduler on real production workloads?
Phase 2
Upstream Controller Comparison
Deploy the governing engine alongside the default Kubernetes scheduler in a shadow mode: both schedulers receive the same scheduling requests, both produce decisions, only the default scheduler's decisions are executed. The governing engine's decisions are logged and compared. This produces a Class P dataset without requiring the governing engine to control production traffic — the risk profile is identical to running a logging agent.
Phase 3
Canary Deployment
Route a defined percentage of scheduling decisions through the governing engine on a production cluster. Measure throughput, latency, and cost against the control group. This is the first true Class P result: the governing engine controlling real production traffic, measured against a live baseline. The canary percentage starts at 1% and increases as confidence accumulates.
Phase 4
Multi-Vendor Hardware Comparison
Run the sealed benchmark harness on AWS, GCP, and Azure instances of equivalent specification. Compare the throughput gains across vendors. The renormalization group fixed-point theorem predicts that the gains will be consistent across hardware vendors — this is the test of that prediction. Any significant divergence would be scientifically interesting and would be published as a finding, not suppressed.
WHY THIS PAGE EXISTS
Engineers hate hidden caveats.
THE STANDARD PRACTICE
The standard practice in enterprise software benchmarking is to publish the best number, bury the methodology in an appendix, and hope that no one reads the appendix carefully. This practice works until an engineer does read the appendix — at which point the vendor loses the engineer's trust permanently, along with every engineer that engineer talks to.
THE ALTERNATIVE
State the evidence class in the first paragraph. Define what was measured and what was not. Publish the fingerprint. Publish the methodology. Publish the raw data. Give engineers a one-command verification entry point. Invite them to run the next validation steps and publish their results. This approach loses the engineers who were never going to trust the numbers anyway. It gains the engineers who matter: the ones who will clone the repository, run the harness, and write the next chapter.
THE MOAT
Transparency is not a concession. It is a moat. A vendor who publishes sealed fingerprints, defines evidence classes, and invites independent verification has made a commitment that cannot be faked retroactively. A vendor who buries caveats can always claim the fine print was there. A vendor who puts the caveat in the first paragraph has made the caveat the product. The product is: we will tell you exactly what we know and exactly what we do not know, and we will give you the tools to find out for yourself.
THE SEAL SPECIFICATION
What SEAL.json Contains
| FIELD | VALUE | MEANING |
|---|---|---|
| fingerprint | 66c8f2d94b3e4725 | SHA-256 prefix of the compiled C++ governor binary at the time of the sealed run. |
| coreSha | bd615f15 | Git commit SHA of omni_controller/controller.cpp at the time of the sealed run. |
| sealed_at | 2026-10-25T17:07:20+00:00 | ISO 8601 timestamp of the sealed run, recorded by the GitHub Actions runner. |
| twin_pairs | 17 | Number of sealed twin-arm experiment pairs. Each pair runs the baseline and governed policies on the same hardware in the same thermal state. |
| paired_reps | 97 | Number of paired repetitions per twin pair. Each repetition is an independent scheduling episode. |
| monte_carlo_samples | 10,000,000 | Number of Monte Carlo samples used to construct the confidence intervals on the sealed results. |