--- title: Agent labor markets are evaluation markets slug: agent-labor-markets-are-evaluation-markets date: 2026-08-09 updated: 2026-08-09 author: Kai Kim status: published confidence: 0.6 tags: [ai agents, marketplaces, evaluation, market design, quality control] categories: [hypotheses] summary: When the supplier is software, the marginal cost of producing one more plausible deliverable falls to almost zero. Labor stops being the scarce input and evaluation becomes the scarce input. My hypothesis is that a marketplace for AI agent work is therefore not a matching market at all: it is an evaluation market, and the platform that wins will be the one that makes deciding which submission is correct cheap, fast and trustworthy. Everything else, discovery, payment, settlement, is already commodity infrastructure. key_takeaways: - When submissions cost the supplier almost nothing, abundance moves from the labor side to the output side and the buyer's attention becomes the binding constraint. - A marketplace built for scarce labor spends its design budget on matching and bidding, which is the wrong budget for a market with unlimited plausible supply. - Plausibility is the specific danger: agent output is confident and well formatted whether or not it is correct, so surface quality stops carrying information. - Prior labor markets that industrialized dispatch, from piecework to crowd work platforms, all converged on redundancy, gold standard tasks and reputation weighting to solve the same problem. - The falsifier is clean: if buyers happily absorb review cost and ask instead for better matching, this hypothesis is wrong. reading_time: related: [birdfury-open-mission-network] changelog: - "2026-08-09 | 0.60 | Initial publication | Extracted the general claim from the Birdfury experiment so it can be tested independently of that product" --- # Question If AI agents can supply an unlimited amount of plausible work, what is the scarce resource in a market for that work, and therefore what is the market actually selling? # Observation Two observations sit behind this document, one from building and one from reading. From building: I ran an open mission network where any human or autonomous agent could submit work against a paid task without a claim step. The design assumed that removing friction from supply would produce a market. It did not. The full record, including the numbers that argue against my original reasoning, is in [[birdfury-open-mission-network]]. From reading the wider landscape: the infrastructure layer for agent commerce converged fast. Machine readable payment terms over HTTP, stablecoin settlement in seconds for a fraction of a cent, gasless transfer authorizations, an open tool protocol for discovery. Each of these was a research topic in one year and a default in the next. What did not converge is any accepted way for a buyer to know whether an agent's output is correct before paying for it, or who is accountable when it is not. A third observation, smaller but sharp: agent output fails differently than human output. Weak human work usually looks weak. Weak agent work looks finished. It is well formatted, confidently worded, complete in structure and wrong in substance. The usual cheap signals a buyer uses to triage, does it look serious, is it well presented, did they follow the brief's format, have stopped carrying information. # Hypothesis A marketplace for AI agent work is not a matching market. It is an evaluation market, and its defensible value is created at the moment a buyer decides which submission is correct. Stated so it can be falsified: in agent labor markets, the buyer's cost of evaluating submissions will exceed the buyer's cost of finding suppliers by a wide enough margin that platforms competing on matching, routing, ranking or discovery will lose to platforms competing on evaluation, verification and accountability. Confidence 0.6. I hold it more strongly than the evidence strictly supports, because I have one negative result of my own and a set of historical analogies, not a controlled comparison. # Why The causal argument is a cost structure argument. In a human labor market, producing a deliverable is expensive for the supplier. That expense does useful work for the buyer: it rations supply, so each proposal that arrives carries a costly signal of intent, and the buyer can spend its scarce attention on matching rather than on filtering. Bidding, ranking, and reputation exist to solve the question "who should do this", because doing it is the expensive part. When the supplier is software, that expense collapses. One agent can answer forty briefs in the time a person reads one. Rationing disappears, the costly signal disappears with it, and the buyer inherits the filtering job that scarcity used to do for free. The market's centre of gravity moves from "who should do this" to "which of these is right", and those are different products with different data models. The first needs profiles, prices and ranking. The second needs acceptance criteria, verification, redundancy, provenance and a way to hold someone accountable. There is a second mechanism reinforcing the first. Because plausible output is cheap to produce, adversarial and lazy supply is also cheap. A market that cannot distinguish correct from plausible does not merely inconvenience buyers, it selects for suppliers optimized to pass shallow review. That is a degenerative loop, and it ends with buyers leaving. Evaluation quality is therefore not a feature of such a market, it is the condition for the market existing at all. # Supporting Evidence - Crowd work platforms that industrialized dispatch, most visibly Amazon Mechanical Turk, discovered that dispatch was the easy half. Requesters independently converged on the same countermeasures: redundant assignment of the same task to multiple workers, gold standard items with known answers seeded into the stream, and reputation weighting. All three are evaluation mechanisms, and none of them are matching mechanisms. - Data labeling as an industry priced itself on quality assurance, not on labeling. The consensus pipeline, multiple annotators plus adjudication, is an evaluation market wearing a labor market's clothes. - Open source bounty programs, which most resemble what I built, keep a claim or assignment step even though it slows throughput. The claim step is best read as an evaluation subsidy: by rationing who is allowed to work, the maintainer avoids reviewing many parallel attempts. - Software engineering has already run this experiment internally. When code generation made producing a patch nearly free, the constraint moved to review capacity, and the tooling investment followed it there rather than into generating more patches. - My own negative result: a claim free, fully open, payment ready market that optimized the supply side produced approximately zero demand ([[birdfury-open-mission-network]]). # Counter Arguments The strongest objection is that evaluation may not stay expensive. If model capability keeps improving, correctness could become the default and verification could become a formality, in which case the market reverts to a matching market and my hypothesis ages badly. I find this less persuasive than it sounds, because verification cost tracks task ambiguity rather than model strength, and ambiguous tasks are exactly the ones worth outsourcing. However, I cannot rule it out for well specified, mechanically checkable work such as code with a test suite, and that category is large and growing. A second objection: buyers may simply refuse to run an evaluation market. They may prefer a managed outcome, one accountable vendor who absorbs the review burden and carries liability. If that preference dominates, the winner is not an evaluation platform but an agency, and marketplaces of any design lose. This is the objection I take most seriously, because it is what buyers historically chose whenever quality was hard to observe. A third objection: evaluation may not be defensible even if it is valuable. Verification could commoditize into an open protocol the way payment did, in which case being good at it earns no durable position. The rebuttal is that evaluation depends on accumulated outcome history, which is a data asset rather than a protocol, but that rebuttal is an assumption and not evidence. # Current Conclusion The scarce resource in an agent labor market is the buyer's confident yes. Everything upstream of that moment, discovery, routing, negotiation, settlement, has become commodity infrastructure that any competitor can assemble in weeks. Everything downstream of it, payment and accounting, was solved by stablecoin rails. So the design question I now consider correct is not "how do I connect the right agent to the right task", it is "how do I make a buyer confident in an output produced by a counterparty it cannot inspect, cheaply enough that it does the deal again next week". Redundancy, structured acceptance criteria, outcome history, and someone to hold accountable are the raw materials. A marketplace that treats these as trust and safety chores has misread which part of its stack is the product. # Unknowns - How much does evaluation actually cost per submission, in buyer minutes, for realistic tasks? I have an argument and no measurement. - Which tasks are mechanically verifiable enough to escape this hypothesis entirely, and how large is that category? - Does accumulated outcome history genuinely compound into defensibility, or does it decay as fast as models change? - Do buyers prefer a marketplace with strong evaluation, or an agency that removes evaluation from their plate completely? - Is redundant parallel submission actually cheaper for the buyer than a single trusted supplier, once review time is priced honestly? # What Would Change My Mind Upward, toward 0.8 or higher: - A buyer who explicitly pays more for a platform because of its verification layer, while treating supplier discovery as interchangeable. - Measurable review time as the dominant line item in a buyer's cost of using agent labor. - A platform that wins share primarily on evaluation tooling rather than on liquidity or price. Downward, toward 0.3 or lower: - Buyers reporting that review is trivial and asking instead for better matching, filtering or discovery. - Automated verification becoming so reliable and so commoditized that no platform can differentiate on it. - Agencies and managed vendors capturing the demand outright, which would mean the marketplace form is wrong regardless of which layer it optimizes. # Next Research - Instrument evaluation cost directly: give a buyer several submissions for a real task and measure the minutes to a confident decision. - Map the task space by verifiability, from test covered code at one end to strategy memos at the other, and locate where outsourcing demand actually concentrates. - Study how consensus pipelines in data labeling priced and structured adjudication, and test which mechanisms transfer to open submission. - Interrogate the agency counter hypothesis on its own terms rather than as a footnote to this one. # Implications For markets: value in agent commerce accrues to whoever owns the moment of acceptance, not the moment of matching. That is where the data, the accountability and the recurring reason to return all live. For builders: if you are building an agent marketplace, your roadmap is probably inverted. Discovery, profiles and routing feel like the product and are the commodity. Acceptance criteria, redundancy, outcome history and dispute handling feel like overhead and are the product. For Kaikim Labs: this hypothesis is the reason the next build is not another board. It is testable without a marketplace attached, which makes it a better foundation for the next experiment than the one that produced it. # Related Research - [[birdfury-open-mission-network]] # References 1. Birdfury machine readable status summary: https://birdfury.com/llms.txt 2. Model Context Protocol: https://modelcontextprotocol.io 3. x402 payment protocol: https://www.x402.org