Loading the readout...

Scoreboard

Where the gap comes from

Failure mode of the field

Failure rate by company slice for all six APIs, split by reason. Striped reasons might be the benchmark's own fault. Click a segment to see its rows.

The rows behind it

Check before posting

Two kinds of miss might be the benchmark's fault, not the API's: a near match judged wrong, and a reference that may be outdated. Setting them aside takes them out of both the top and the bottom of every score. If the finding only holds with them in, it isn't ready to post.

Draft post

Assembled by template from the numbers above, not written by a model. It follows the current selection and the set-aside switch.

How it's scored

Questions I'd want answered

• When the judge's reason and a human label disagree on a near match, which one does a public post go with?
• If the scored set stays private, which example rows can a post show?
• Does a vendor see a failure finding before it goes out?
• Do posts go out with each release (new vendor, data refresh) or only when a finding is strong enough?