Improve reconciliation service error handling
With the wonky Wikidata reconciliation service currently, I managed to get a wide variety of errors with the reconciliation process then hanging and not being cancellable. We need to improve this.
Errors logged on OpenRefine console
07:42:26.185 [ refine] POST /command/core/guess-types-of-column (42ms)
07:42:26.416 [..le.refine.model.Project] Loaded project 1943190398909 from disk in 0 sec(s) (230ms)
07:42:42.934 [ refine] POST /command/core/reconcile (16518ms)
07:52:49.606 [..ic.HttpRequestRetryExec] ex-0000000067 recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when sending request to http://localhost:8000;request will be automatically re-executed in 0 MILLISECONDS (exec count 2) (606671ms)
07:53:40.922 [..ic.HttpRequestRetryExec] ex-0000000084 recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when sending request to http://localhost:8000;request will be automatically re-executed in 0 MILLISECONDS (exec count 2) (51315ms)
07:56:25.254 [..e.io.FileProjectManager] Saving some modified projects ... (164332ms)
07:56:25.686 [ project_utilities] Saved project '1943190398909' (432ms)
08:18:54.243 [ refine] GET /command/core/get-csrf-token (1348556ms)
08:18:54.334 [ refine] POST /command/core/cancel-processes (90ms)
08:18:54.782 [..le.refine.model.Project] Loaded project 1943190398909 from disk in 0 sec(s) (448ms)
08:18:54.860 [ refine] GET /command/core/get-history (77ms)I'm not sure where the NoHttpResponse exceptions came from, but they might have been involved in killing the reconciliation task.
Although the cancel-processes command seems to have mostly worked, the front end was still spinning with the "Canceling..." dialog up.
Errors populated in the recon cells included:
- No. of recon objects was less than no. of jobs
- HTTP error 403 : for URL /en/api [sic] - this should be fixed to return the full path with query = The reconciliation service returned an invalid response
I think the first two are variations due to how the calls are batched with the underlying error being a 403 in both cases.
The middle one might have been from this error that the reconciliation service got:
reconcile-1 | File "/openrefine-wikibase/wdreconcile/engine.py", line 57, in _srsearch
reconcile-1 | resp = await r.json()
reconcile-1 | File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 1109, in json
reconcile-1 | headers=self.headers,
reconcile-1 | aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.wikidata.org/w/api.php?action=query&format=json&list=search&srnamespace=0&srlimit=50&srsearch=Susan+Mallery&srwhat=text')
where it attempted a JSON decode before checking the response status code.To Reproduce
Steps to reproduce the behavior:
- First, do ...
- Then, ...
- Finally, ...
Current Results
Expected Behavior
Screenshots
Versions
- Operating System:
- Browser Version:
- JRE or JDK Version:
- OpenRefine:
Datasets
Additional context
Source: OpenRefine/OpenRefine