A benchmark map of the US economic system
Our estimates of complete financial protection, and of patterns in protection, all come from the preliminary listing of 29 benchmarks we selected. This just isn’t an entire or completely consultant pattern of the broader universe of unsaturated, public, agentic financial benchmarks. Any incomplete listing will essentially bias down estimates of financial protection, and it’s troublesome to certain that bias. We wish to stress, subsequently, that these outcomes are provisional. We will replace them as we increase the preliminary benchmark listing.
We additionally capped the variety of included benchmarks in every job household to be able to maximize breadth of protection. We suppose this makes our estimate of economy-wide benchmark protection extra correct, at the price of biasing down our estimates of protection within the most-benchmarked job households (e.g. software program engineering).
By choosing for unsaturated benchmarks, we threat understating benchmark protection on O*NET duties that are simpler for LLMs to finish. On the opposite hand, together with saturated benchmarks would threat overstating benchmark protection if saturation is downstream of decrease activity realism. The latter declare appears doubtless, provided that benchmarks typically overstate LLM efficiency in real-world deployments. We suppose that the bias from together with saturated benchmarks could be larger than the bias from excluding them, however neither choice is right.
Finally, whereas we define our benchmark choice standards within the Benchmark Index tab, a number of standards (like “Economic Realism”) are troublesome to formalize and introduce some subjectivity in our selections.
