Fresh installs cannot publish Connector Builder

Topic

Can't deploy connectors

Relevant information

Fresh installs cannot publish Connector Builder connectors: baked CDK seed stops at major 4, and the DockerHub refresh fails on paginated requests

Platform Version

2.2.0 (abctl / kind on a Linux VM). The mechanism is version-independent — reproduced against a 2.2.0 install; the seed is identical on older versions.

What step the error happened?

Other — publishing a Connector Builder connector.

Summary

On a fresh install, POST /api/v1/connector_builder_projects/publish returns 500 for any manifest declaring a CDK version above major 4. The same manifest resolves and tests fine.

Two separate problems combine:

1. The baked declarative_manifest_image_version seed stops at major 4, while the shipped manifest-server is 7.23.7.
2. DeclarativeSourcesUpdater, which is supposed to fill the gap from DockerHub, fails on every run — so the table never advances past the seed.

Reproduction

1. Fresh abctl local install (2.2.0).
2. Create a Connector Builder connector with a manifest declaring version: 7.x.
3. Resolve / test it — works.
4. Publish — 500.

select major_version, image_version from declarative_manifest_image_version order by major_version;
 major_version | image_version
---------------+---------------
             0 | 0.90.0
             1 | 1.7.0
             2 | 2.1.0
             3 | 3.10.4
             4 | 4.3.0
(5 rows)

No row for major 7 → the publish path throws on the lookup.

Problem 1: the seed is three majors behind

A fresh install ships mappings only through major 4, but the Connector Builder produces major-7 manifests. So a fresh install cannot publish anything it can build until the cron succeeds at least once.

This is exactly the failure mode anticipated in #38575, including the symptom:

you can build and test a connector without dockerhub access… you just can't save it.

That issue was closed, and the seed did move from majors 0–1 to 0–4 — but the CDK has since reached 7 and the seed hasn't kept pace.

Problem 2: the DockerHub refresh cannot complete

RemoteDeclarativeManifestImageVersionsProvider (per #53665) paginates the full tag list at page_size=100 and throws on the first non-2xx:

var nextUrl: String? = "https://hub.docker.com/v2/repositories/$repository/tags?page_size=100"
while (nextUrl != null) {
  ...
  if (!response.isSuccessful || response.body == null) {
    throw IOException("Unexpected response from DockerHub API: ${response.code} ${response.message}")
  }

airbyte/source-declarative-manifest now has 1002 tags → 11 requests per run.

DockerHub's anonymous allowance for this endpoint is lower than that. Measured from a single IP:

page 1–10   200
page 11     403
page 12+    403

Ten allowed, eleven required. The run cannot complete, and because any non-2xx throws, the ten successful pages are discarded — the table is never partially updated.

Cron log, every 10 minutes since install:

INFO  i.a.c.j.DeclarativeSourcesUpdater(updateDefinitions) - Getting latest CDK versions and updating declarative sources...
INFO  i.a.c.i.RemoteDeclarativeManifestImageVersionsProvider(getTagsAndShasForRepository) - Fetching image tags and SHAs for airbyte/source-declarative-manifest...
ERROR i.m.s.DefaultTaskExceptionHandler(handle) - Error invoking scheduled task for bean
      [io.airbyte.cron.jobs.DeclarativeSourcesUpdater@...] Unexpected response from DockerHub API: 403
        at io.airbyte.config.init.RemoteDeclarativeManifestImageVersionsProvider.getTagsAndShasForRepository(RemoteDeclarativeManifestImageVersionsProvider.kt:63)
        at io.airbyte.config.init.RemoteDeclarativeManifestImageVersionsProvider.getLatestDeclarativeManifestImageVersions(RemoteDeclarativeManifestImageVersionsProvider.kt:30)
        at io.airbyte.config.init.DeclarativeSourceUpdater.apply(DeclarativeSourceUpdater.kt:40)
        at io.airbyte.cron.jobs.DeclarativeSourcesUpdater.updateDefinitions(DeclarativeSourcesUpdater.kt:49)

Since the allowance is keyed per IP and shared, whether an install survives 11 rapid requests is effectively luck. Two 2.2.0 installs on different networks, same day: one completed a full refresh, the other has failed every run since install. As the tag count grows this gets worse for everyone.

Impact

• Any fresh self-hosted install on an IP with prior DockerHub traffic — notably cloud IPs, which are shared — cannot publish Connector Builder connectors at all.
• The failure is silent and hard to diagnose: the publish 500 has an empty body and logs no stack trace, since the throw happens in a path with no error handling. The only signal is a cron log nobody has reason to read.
• Installs that worked for months break on rebuild, because the working state was a successful cron run that can no longer be reproduced.
• This also breaks Terraform's airbyte_declarative_source_definition. The create path hits the same lookup, so it writes a draft, fails to activate it, and reports success — leaving silent drafts with no error.

Suggested fixes

The rate limit isn't yours to control, but these are:

1. Refresh the baked seed to the current CDK major and keep it current at release time, so a fresh install can publish what its own builder produces without any network call.
2. Don't discard partial results. Accumulate per page and commit what was fetched rather than throwing away ten good pages because the eleventh failed.
3. Stop paginating the whole tag list. Only the newest tag per major is needed; walking 1002 tags to find ~8 values is what puts the job over the allowance.
4. Surface the failure. A 500 with an empty body on publish should at minimum log why, and ideally return "no image mapping for CDK major N."
5. Optionally, allow DockerHub credentials for this call so the counter moves off the shared IP.

Workaround

Insert the missing rows manually, copying image_version and image_sha from an install where the cron succeeded:

insert into declarative_manifest_image_version
  (major_version, image_version, created_at, updated_at, image_sha)
values (7, '7.23.8', now(), now(), 'sha256:<digest>');

Publishing works immediately afterward, with no restart.

Internal Tracking: https://github.com/airbytehq/oncall/issues/13360