Support connecting to BrightData Scraping Browsers
I'd like to use scalable browser infrastructure services, such as the BrightData Scraping Browser, which integrate well with Puppeteer. But, I have encountered some issues when trying to use these services with go-rod. I would like to request the following enhancements to improve compatibility:
1. WebSocket Authentication:
The connection to these services’ WebSocket requires authentication (e.g., wss://user:pass@host:9222). However, go-rod does not currently send the necessary authentication headers, which I think are not defined on any WebSocket standard.
Through my research, I discovered that authentication is performed using Basic tokens. I have implemented a working solution to inject the Authorization header. However, I am unsure if this is the optimal place to inject it. If this solution aligns with the project's direction, I am willing to submit a PR with my implementation.
2. Less Restrictive WebSocket Response Handling:
The services sometimes send responses that deviate from the expected go-rod Response structure, causing panics due to unmarshalling failures. Specifically, the Error struct expects an integer Code, but some responses include a string (e.g., "navigate_limit").
The Response structure is defined this way:
type Response struct {
ID int `json:"id"`
Result json.RawMessage `json:"result,omitempty"`
Error *Error `json:"error,omitempty"`
}
type Error struct {
Code int `json:"code"`
Message string `json:"message"`
Data string `json:"data"`
}And Brightdata is sending to me struct that panic, like this
{
"id": 27,
"sessionId": "BRD_461626884EEF95862B6188C2DBB766D1",
"error": {
"message": "Page.navigate limit reached",
"code": "navigate_limit" // This is expected as an Int.
},
"duration": 1.2261550000112038
}To make go-rod more compatible, it may be necessary to relax the strictness of the standard for the Error struct. I would appreciate guidance on the best approach to achieve this flexibility. If you agree with this, I am happy to work on the implementation with a bit of guidance.
Source: go-rod/rod