Is Your AI Forecaster Copying the Market?

An AI system can return a plausible probability, explain it fluently, and still leave a developer unsure what has been measured. Did the model assess the evidence, or did it repeat a probability supplied by a prediction market? A useful test starts by separating those tasks. Give the same model the same question and evidence, then compare runs with and without the market estimate.

In its October 2025 ForecastBench analysis, the Forecasting Research Institute reported that GPT-4.5 reproduced supplied market probabilities exactly on 26 of 122 questions. That finding concerned a particular model and evaluation condition. It does not establish that every matching forecast was copied, or that all AI forecasters behave this way, but it is a very useful starting point.

The researchers maintained separate leaderboards for models with different access to information and tools. For a developer, the question is what the application is supposed to do: form its own estimate or interpret an existing market view. That decision determines which inputs belong in the test.

Market activity and forecasting quality need separate measures. 2026 World Cup trading volume across Polymarket and Kalshi exceeded $5 billion during the tournament’s first two weeks, according to AlphaWire reporting on27 June 2026. That figure describes trading activity; assessing an AI forecast still requires a defined set of questions, recorded probabilities, timestamps, and eventual outcomes.

Separate agreement from accuracy

A market probability is an input when the model can read it and a comparison point when it is withheld. Those roles should remain visible in the evaluation record. Otherwise, an apparently strong result can obscure how much information the system received.

For an independent forecast, ask the model to estimate the event’s probability using an evidence pack that excludes market probabilities. For a market-assisted forecast, supply the same material plus the recorded market estimate. Both tasks can be useful, but they answer different questions about the application.

Agreement and accuracy also require separate checks. Agreement concerns how closely the model’s output resembles the market estimate. Accuracy concerns how the forecast relates to the eventual outcome across the questions tested. A difference from the market is not automatically a better prediction, and a match is not automatically evidence of copying.

Before running either condition, write down what would count as the event happening. Preserve the market’s exact question, deadline, and resolution rules. A model answering a broader question may produce a reasonable explanation that cannot be assessed against the intended outcome.

Change one input

Build the comparison around paired runs for each question. Keep the model version, instructions, supplied evidence, and generation settings fixed. Change only access to the market probability.

Use the market observation available at the chosen information cutoff, rather than a later value. Save its timestamp alongside the forecast. Comparing a model’s earlier estimate with a market observation collected after new information appeared would introduce another difference into the test.

Check the evidence pack before calling the first condition independent. A news report can quote market odds even when no market page is supplied. Remove those references consistently, and use that same edited evidence in both conditions. Otherwise, withholding one explicit number may leave another route to it.

Avoid unrestricted browsing in this controlled comparison. If the model can retrieve market probabilities elsewhere, the intended difference between conditions disappears. A separate evaluation can examine the application with browsing enabled, provided its retrieved material is recorded, and its capabilities are described accurately.

Keep the comparison repeatable

Historical tests also need protection against later information. Preserve copies of documents available before the chosen cutoff, rather than relying on live pages that may have changed. Keep outcome records separate from material the forecasting model receives.

Document restrictions cannot erase information already present in a model’s training. For a cleaner test of future forecasting, record predictions while events remain unresolved and assess them afterwards. Historical evaluations should state what they controlled and what they could not establish.

Run each condition in a fresh session. Anthropic’s January 2026 guidance on evaluating AI agents recommends isolated trials and explains how leftover files or cached information can distort results. An independent run should not inherit the market estimate from an earlier assisted run through conversation history or shared storage.

Repeat both conditions across questions and repeated attempts, using the same number of attempts. Record the probability change in percentage points and whether the assisted forecast moved towards the supplied market value. Compare these shifts with the variation between repeated runs under each condition. Keep every output, including missing or invalid probabilities, instead of selecting the most convincing answers.

Keep explanations for inspection. A persuasive account does not prove how the probability was produced. Check whether cited evidence appeared in the supplied documents and whether the reasoning addresses the resolution rules.

If supplied probabilities shift outputs beyond their usual variation, the results indicate sensitivity to that information under the recorded conditions. The comparison cannot reveal every step of the model’s reasoning. If outputs stay similar, shared evidence or previously learned information may still explain the agreement. Report the inputs, observed changes, and unresolved limits together, so an independent forecast is not confused with an assisted one.