Marketplace search ranking is not a single number

Ask the VS Code Marketplace the same search twice, a few seconds apart, and the top nine results usually stay put. Positions 10 through 100 shuffle. That is not your extension rising and falling. It is the gallery returning a different permutation of the same relevance band.

VSXRank did not reverse-engineer Microsoft’s ranker. We measured what the unofficial gallery API actually returns when you POST the same searchText with sortBy: 0 (relevance). Open VSX, asked the same query, is stable between calls.

What we do with that

For every keyword a customer tracks we hit the Marketplace three times and store the full ordered list each time, out to position 1,000. The number on the dashboard is the median of those three samples. The chart can show a min/max band so a one-sample jump does not look like a trend.

Alerts only fire when the median moves by about five positions. That is wider than the shuffle we see in the 10–100 band, so a “you dropped eight places” email is a real move, not a redraw. Open VSX ordering does not shuffle, so its threshold can be lower; we still use the same product rule so the two columns stay comparable.

If a sample does not find the extension inside the window, that sample is missing. A median built from two hits and a miss is optimistic — the UI will say so rather than pretend we saw it three times.

Relevance is not installs

A 3,500-install extension will outrank a 10,000-install one on a query that matches its name more tightly. We watched this on our own listing: a smaller competitor sat above us on the phrase that is our display name because the gallery weights exact token overlap harder than install count.

Consequences:

• Head terms are mostly unwinnable. “python”, “docker”, “git” are occupied by Microsoft and the two or three names everyone already types. Daily tracking on those words is a chart of other people’s weather.
• The keywords that move are the specific ones. Product name, adjacent tool, the job the README actually describes. Those are the queries where a README change, a display-name tweak or a competitor’s launch shows up inside a week.
• Total result count is a crowdedness signal. “claude code” returned 37,861 Marketplace hits on 5 September 2026. Ranking #292 there is a different sport from ranking #8 in a 12-result query.

We store the total alongside the position. A slide from #40 to #55 in a 200-result query is not the same event as the same slide in a 20,000-result query.

Why the window is 1,000, not 100

Most mid-size extensions live below #100 on the words they actually care about. On 5 September 2026, agsoft.claude-history-viewer was #292 of 37,861 for “claude code” — invisible to any tracker that stops at 100, and still well inside a 1,000-deep sample. Past 1,000 we render >1,000 rather than invent a position. Why we track to 1,000 is the longer version.

VSCodeRank, the only other paid tracker, samples positions 1 to 100. That is the upsell, not a secret algorithm.

What to do with this as a publisher

1. Track the five to ten queries a user would type after they already know the category, not the category word itself.
2. Ignore day-to-day jitter inside five places on the Marketplace. Look at the median over a week.
3. Treat a competitor who outranks you on your own name as a naming problem, not an install-count problem.
4. Keep Open VSX in the same table. The order there is stable, the audience is different, and a jump on Open VSX is usually a release, not a ranker mood. See publishing to Open VSX.

Add the keywords on a watchlist entry. We crawl each unique keyword once for everyone who tracks it, three Marketplace samples included, so the cost of a new phrase is one search, not one search per customer.