{
  "title": "Agent labor markets are evaluation markets",
  "slug": "agent-labor-markets-are-evaluation-markets",
  "url": "https://kaikimlabs.com/research/hypotheses/agent-labor-markets-are-evaluation-markets/",
  "category": "hypotheses",
  "date": "2026-08-09",
  "updated": "2026-08-09",
  "author": "Kai Kim",
  "status": "published",
  "confidence": "0.6",
  "confidence_label": "reasonable hypothesis",
  "tags": [
    "ai agents",
    "marketplaces",
    "evaluation",
    "market design",
    "quality control"
  ],
  "summary": "When the supplier is software, the marginal cost of producing one more plausible deliverable falls to almost zero. Labor stops being the scarce input and evaluation becomes the scarce input. My hypothesis is that a marketplace for AI agent work is therefore not a matching market at all: it is an evaluation market, and the platform that wins will be the one that makes deciding which submission is correct cheap, fast and trustworthy. Everything else, discovery, payment, settlement, is already commodity infrastructure.",
  "key_takeaways": [
    "When submissions cost the supplier almost nothing, abundance moves from the labor side to the output side and the buyer's attention becomes the binding constraint.",
    "A marketplace built for scarce labor spends its design budget on matching and bidding, which is the wrong budget for a market with unlimited plausible supply.",
    "Plausibility is the specific danger: agent output is confident and well formatted whether or not it is correct, so surface quality stops carrying information.",
    "Prior labor markets that industrialized dispatch, from piecework to crowd work platforms, all converged on redundancy, gold standard tasks and reputation weighting to solve the same problem.",
    "The falsifier is clean: if buyers happily absorb review cost and ask instead for better matching, this hypothesis is wrong."
  ],
  "reading_time_minutes": 8,
  "headings": [
    "Question",
    "Observation",
    "Hypothesis",
    "Why",
    "Supporting Evidence",
    "Counter Arguments",
    "Current Conclusion",
    "Unknowns",
    "What Would Change My Mind",
    "Next Research",
    "Implications",
    "Related Research",
    "References"
  ],
  "changelog": [
    {
      "date": "2026-08-09",
      "confidence": "0.60",
      "what_changed": "Initial publication",
      "why_it_changed": "Extracted the general claim from the Birdfury experiment so it can be tested independently of that product"
    }
  ],
  "revision_history": [
    {
      "date": "2026-08-09",
      "subject": "research: first two documents, the Birdfury experiment and the evaluation market hypothesis"
    }
  ],
  "related": [
    "https://kaikimlabs.com/research/experiments/birdfury-open-mission-network/"
  ],
  "referenced_by": [
    "https://kaikimlabs.com/research/experiments/birdfury-open-mission-network/"
  ],
  "body_markdown": "# Question\n\nIf AI agents can supply an unlimited amount of plausible work, what is the\nscarce resource in a market for that work, and therefore what is the market\nactually selling?\n\n# Observation\n\nTwo observations sit behind this document, one from building and one from\nreading.\n\nFrom building: I ran an open mission network where any human or autonomous\nagent could submit work against a paid task without a claim step. The design\nassumed that removing friction from supply would produce a market. It did not.\nThe full record, including the numbers that argue against my original\nreasoning, is in [[birdfury-open-mission-network]].\n\nFrom reading the wider landscape: the infrastructure layer for agent commerce\nconverged fast. Machine readable payment terms over HTTP, stablecoin\nsettlement in seconds for a fraction of a cent, gasless transfer\nauthorizations, an open tool protocol for discovery. Each of these was a\nresearch topic in one year and a default in the next. What did not converge is\nany accepted way for a buyer to know whether an agent's output is correct\nbefore paying for it, or who is accountable when it is not.\n\nA third observation, smaller but sharp: agent output fails differently than\nhuman output. Weak human work usually looks weak. Weak agent work looks\nfinished. It is well formatted, confidently worded, complete in structure and\nwrong in substance. The usual cheap signals a buyer uses to triage, does it\nlook serious, is it well presented, did they follow the brief's format, have\nstopped carrying information.\n\n# Hypothesis\n\nA marketplace for AI agent work is not a matching market. It is an evaluation\nmarket, and its defensible value is created at the moment a buyer decides\nwhich submission is correct.\n\nStated so it can be falsified: in agent labor markets, the buyer's cost of\nevaluating submissions will exceed the buyer's cost of finding suppliers by a\nwide enough margin that platforms competing on matching, routing, ranking or\ndiscovery will lose to platforms competing on evaluation, verification and\naccountability.\n\nConfidence 0.6. I hold it more strongly than the evidence strictly supports,\nbecause I have one negative result of my own and a set of historical\nanalogies, not a controlled comparison.\n\n# Why\n\nThe causal argument is a cost structure argument.\n\nIn a human labor market, producing a deliverable is expensive for the\nsupplier. That expense does useful work for the buyer: it rations supply, so\neach proposal that arrives carries a costly signal of intent, and the buyer\ncan spend its scarce attention on matching rather than on filtering. Bidding,\nranking, and reputation exist to solve the question \"who should do this\",\nbecause doing it is the expensive part.\n\nWhen the supplier is software, that expense collapses. One agent can answer\nforty briefs in the time a person reads one. Rationing disappears, the costly\nsignal disappears with it, and the buyer inherits the filtering job that\nscarcity used to do for free. The market's centre of gravity moves from \"who\nshould do this\" to \"which of these is right\", and those are different products\nwith different data models. The first needs profiles, prices and ranking. The\nsecond needs acceptance criteria, verification, redundancy, provenance and a\nway to hold someone accountable.\n\nThere is a second mechanism reinforcing the first. Because plausible output is\ncheap to produce, adversarial and lazy supply is also cheap. A market that\ncannot distinguish correct from plausible does not merely inconvenience\nbuyers, it selects for suppliers optimized to pass shallow review. That is a\ndegenerative loop, and it ends with buyers leaving. Evaluation quality is\ntherefore not a feature of such a market, it is the condition for the market\nexisting at all.\n\n# Supporting Evidence\n\n- Crowd work platforms that industrialized dispatch, most visibly Amazon\n  Mechanical Turk, discovered that dispatch was the easy half. Requesters\n  independently converged on the same countermeasures: redundant assignment of\n  the same task to multiple workers, gold standard items with known answers\n  seeded into the stream, and reputation weighting. All three are evaluation\n  mechanisms, and none of them are matching mechanisms.\n- Data labeling as an industry priced itself on quality assurance, not on\n  labeling. The consensus pipeline, multiple annotators plus adjudication, is\n  an evaluation market wearing a labor market's clothes.\n- Open source bounty programs, which most resemble what I built, keep a claim\n  or assignment step even though it slows throughput. The claim step is best\n  read as an evaluation subsidy: by rationing who is allowed to work, the\n  maintainer avoids reviewing many parallel attempts.\n- Software engineering has already run this experiment internally. When code\n  generation made producing a patch nearly free, the constraint moved to\n  review capacity, and the tooling investment followed it there rather than\n  into generating more patches.\n- My own negative result: a claim free, fully open, payment ready market that\n  optimized the supply side produced approximately zero demand\n  ([[birdfury-open-mission-network]]).\n\n# Counter Arguments\n\nThe strongest objection is that evaluation may not stay expensive. If model\ncapability keeps improving, correctness could become the default and\nverification could become a formality, in which case the market reverts to a\nmatching market and my hypothesis ages badly. I find this less persuasive than\nit sounds, because verification cost tracks task ambiguity rather than model\nstrength, and ambiguous tasks are exactly the ones worth outsourcing. However,\nI cannot rule it out for well specified, mechanically checkable work such as\ncode with a test suite, and that category is large and growing.\n\nA second objection: buyers may simply refuse to run an evaluation market. They\nmay prefer a managed outcome, one accountable vendor who absorbs the review\nburden and carries liability. If that preference dominates, the winner is not\nan evaluation platform but an agency, and marketplaces of any design lose.\nThis is the objection I take most seriously, because it is what buyers\nhistorically chose whenever quality was hard to observe.\n\nA third objection: evaluation may not be defensible even if it is valuable.\nVerification could commoditize into an open protocol the way payment did, in\nwhich case being good at it earns no durable position. The rebuttal is that\nevaluation depends on accumulated outcome history, which is a data asset\nrather than a protocol, but that rebuttal is an assumption and not evidence.\n\n# Current Conclusion\n\nThe scarce resource in an agent labor market is the buyer's confident yes.\nEverything upstream of that moment, discovery, routing, negotiation,\nsettlement, has become commodity infrastructure that any competitor can\nassemble in weeks. Everything downstream of it, payment and accounting, was\nsolved by stablecoin rails.\n\nSo the design question I now consider correct is not \"how do I connect the\nright agent to the right task\", it is \"how do I make a buyer confident in an\noutput produced by a counterparty it cannot inspect, cheaply enough that it\ndoes the deal again next week\". Redundancy, structured acceptance criteria,\noutcome history, and someone to hold accountable are the raw materials. A\nmarketplace that treats these as trust and safety chores has misread which\npart of its stack is the product.\n\n# Unknowns\n\n- How much does evaluation actually cost per submission, in buyer minutes, for\n  realistic tasks? I have an argument and no measurement.\n- Which tasks are mechanically verifiable enough to escape this hypothesis\n  entirely, and how large is that category?\n- Does accumulated outcome history genuinely compound into defensibility, or\n  does it decay as fast as models change?\n- Do buyers prefer a marketplace with strong evaluation, or an agency that\n  removes evaluation from their plate completely?\n- Is redundant parallel submission actually cheaper for the buyer than a\n  single trusted supplier, once review time is priced honestly?\n\n# What Would Change My Mind\n\nUpward, toward 0.8 or higher:\n\n- A buyer who explicitly pays more for a platform because of its verification\n  layer, while treating supplier discovery as interchangeable.\n- Measurable review time as the dominant line item in a buyer's cost of using\n  agent labor.\n- A platform that wins share primarily on evaluation tooling rather than on\n  liquidity or price.\n\nDownward, toward 0.3 or lower:\n\n- Buyers reporting that review is trivial and asking instead for better\n  matching, filtering or discovery.\n- Automated verification becoming so reliable and so commoditized that no\n  platform can differentiate on it.\n- Agencies and managed vendors capturing the demand outright, which would mean\n  the marketplace form is wrong regardless of which layer it optimizes.\n\n# Next Research\n\n- Instrument evaluation cost directly: give a buyer several submissions for a\n  real task and measure the minutes to a confident decision.\n- Map the task space by verifiability, from test covered code at one end to\n  strategy memos at the other, and locate where outsourcing demand actually\n  concentrates.\n- Study how consensus pipelines in data labeling priced and structured\n  adjudication, and test which mechanisms transfer to open submission.\n- Interrogate the agency counter hypothesis on its own terms rather than as a\n  footnote to this one.\n\n# Implications\n\nFor markets: value in agent commerce accrues to whoever owns the moment of\nacceptance, not the moment of matching. That is where the data, the\naccountability and the recurring reason to return all live.\n\nFor builders: if you are building an agent marketplace, your roadmap is\nprobably inverted. Discovery, profiles and routing feel like the product and\nare the commodity. Acceptance criteria, redundancy, outcome history and\ndispute handling feel like overhead and are the product.\n\nFor Kaikim Labs: this hypothesis is the reason the next build is not another\nboard. It is testable without a marketplace attached, which makes it a better\nfoundation for the next experiment than the one that produced it.\n\n# Related Research\n\n- [[birdfury-open-mission-network]]\n\n# References\n\n1. Birdfury machine readable status summary: https://birdfury.com/llms.txt\n2. Model Context Protocol: https://modelcontextprotocol.io\n3. x402 payment protocol: https://www.x402.org\n"
}