[BUG]: Confluence import drops code blocks whose macro lists ac:macro-id before ac:name
How are you running AnythingLLM?
Local development
What happened?
The Confluence connector drops the code from a code block when the page stores the code macro with ac:macro-id before ac:name. The text around the block is kept, along with the text of any macro parameters such as a language name, so the imported page shows a bare bash where the code was, or nothing at all when the macro has no parameters. The code should come through as a fenced block, as it does when ac:name comes first.
createDocumentFromPage in collector/utils/extensions/Confluence/ConfluenceLoader/index.js converts code macros with a regex that starts with <ac:structured-macro ac:name="code" (line 111), so it only matches when ac:name is the first attribute. Any other order falls through to html-to-text, which drops the CDATA body.
On master a3557034 I ran createDocumentFromPage on this page body, then again with ac:macro-id="7f3c" moved to the front of the macro tag:
<p>Install with:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="7f3c"><ac:parameter ac:name="language">bash</ac:parameter><ac:plain-text-body><![CDATA[npm install example]]></ac:plain-text-body></ac:structured-macro>ac:name first pageContent="Install with:\n\n```bash\nnpm install example\n```\n"
ac:macro-id first pageContent="Install with:\n\nbash"Real pages are stored this way. In a saved response from https://cwiki.apache.org/confluence/rest/api/content?spaceKey=KAFKA&expand=body.storage,version (the first 25 pages), all 120 code macros put ac:macro-id first, and replaying those page bodies through the loader on master a3557034 produces no code blocks.
This matches the lost code snippets reported in #1501. #1804 closed that issue with the code block conversion, but its regex expects ac:name first.
Are there known steps to reproduce?
- On a self-hosted Confluence instance, find a page whose code macro puts
ac:macro-idbeforeac:name. The content REST API withexpand=body.storageshows the stored markup, which is what the connector reads. - In AnythingLLM, open Data Connectors, choose Confluence, select Self-hosted, enter the instance URL, space key and credentials, and submit the form.
- Open the saved document JSON for that page in the output folder the connector reports. Its
pageContenthas the surrounding text but not the code. If the macro has a language parameter, its name appears where the code should be.
LLM Provider & Model (if applicable)
Not applicable.
Embedder Provider & Model (if applicable)
Not applicable.
Source: Mintplex-Labs/anything-llm