Bug: GoogleSearch YouTube filtering is case-sensitive

Describe the bug

GoogleSearch.search filters YouTube results using a case-sensitive substring comparison.

For example, a result with:

https://youtube.com/watch?v=test

is excluded, while the equivalent valid URL:

https://YouTube.com/watch?v=test

is retained.

URI host names are case-insensitive, so these URLs identify the same host. The filtering behavior should not depend on the casing used in the search result URL.

This can also affect result selection. GoogleSearch.search applies max_results after collecting the unfiltered results. If a mixed-case YouTube result appears before a normal article, it can occupy a limited result slot and cause the article to be omitted.

For this response order:

1. https://YouTube.com/watch?v=test
2. https://example.com/article

with:

max_results=1

the method currently returns the YouTube result instead of the article.

To Reproduce

1. Check out GPT Researcher main at commit:

6f998577d547b1e54ec662dac63583aa11e3b84b

1. Install the project and test dependencies:

python -m pip install -e ".[test]"

1. Create:

tests/test_google_youtube_host_case.py

1. Add this test:

import json
from unittest.mock import MagicMock, patch

import pytest

from gpt_researcher.retrievers.google.google import (
    GoogleSearch,
)


@pytest.mark.parametrize(
    "youtube_host",
    [
        "youtube.com",
        "YouTube.com",
        "YOUTUBE.COM",
    ],
)
def test_google_search_filters_youtube_host_case_insensitively(
    youtube_host,
):
    payload = {
        "items": [
            {
                "title": "Video",
                "link": (
                    f"https://{youtube_host}"
                    "/watch?v=test"
                ),
                "snippet": "video result",
            },
            {
                "title": "Article",
                "link": (
                    "https://example.com/article"
                ),
                "snippet": "article result",
            },
        ]
    }
    response = MagicMock(
        status_code=200,
        text=json.dumps(payload),
    )
    search = GoogleSearch(
        "test query",
        headers={
            "google_api_key": "test-key",
            "google_cx_key": "test-cx",
        },
    )

    with patch(
        "gpt_researcher.retrievers.google.google."
        "requests.get",
        return_value=response,
    ):
        result = search.search(max_results=1)

    assert result == [
        {
            "title": "Article",
            "href": (
                "https://example.com/article"
            ),
            "body": "article result",
        }
    ]

1. Run:

python -m pytest \
  tests/test_google_youtube_host_case.py \
  -q \
  --maxfail=3

1. Observe that the lowercase control passes, while the mixed-case and uppercase hosts fail:

2 failed, 1 passed

Expected behavior

YouTube host filtering should be case-insensitive.

All these equivalent hosts should be filtered:

youtube.com
YouTube.com
YOUTUBE.COM

For every parameterized case, the result should contain the normal article:

[
    {
        "title": "Article",
        "href": "https://example.com/article",
        "body": "article result",
    }
]

The original URL of retained non-YouTube results should remain unchanged.

RFC 3986 section 3.2.2 specifies that the host component is case-insensitive:

https://www.rfc-editor.org/rfc/rfc3986.html#section-3.2.2

The Google Custom Search response schema defines items[].link as the full result URL:

https://developers.google.com/custom-search/v1/reference/rest/v1/Search

Actual behavior

For:

youtube.com

the YouTube result is correctly filtered and the article is returned.

For:

YouTube.com
YOUTUBE.COM

the method returns:

[
    {
        "title": "Video",
        "href": (
            "https://YouTube.com"
            "/watch?v=test"
        ),
        "body": "video result",
    }
]

or its uppercase equivalent.

Because max_results=1, the normal article that follows the YouTube result is omitted.

Representative test summary:

FAILED test_google_search_filters_youtube_host_case_insensitively[YouTube.com]
FAILED test_google_search_filters_youtube_host_case_insensitively[YOUTUBE.COM]

2 failed, 1 passed

Screenshots

Not applicable. This is a deterministic retriever-level reproduction using the public GoogleSearch constructor and its real JSON parsing, filtering, and result-limiting logic.

Only the external Google HTTP response is replaced with a deterministic response.

Desktop (please complete the following information):

• OS: macOS 15.7.3, arm64
• Browser: Not applicable
• GPT Researcher version: 0.14.7
• Revision: main@6f998577d547b1e54ec662dac63583aa11e3b84b
• Python: 3.12.13
• Installation: Source checkout

Smartphone (please complete the following information):

Not applicable.

Additional context

The filtering condition currently compares the complete URL case-sensitively:

link = result.get("link") or ""

if not link or "youtube.com" in link:
    continue

This recognizes:

https://youtube.com/watch?v=test

but not:

https://YouTube.com/watch?v=test

The retained results are later limited:

return search_response[:max_results]

Consequently, a YouTube URL that bypasses the filter can consume a result slot before the slice is applied.

A minimal fix that preserves the existing substring behavior would normalize only for comparison:

if (
    not link
    or "youtube.com" in link.lower()
):
    continue

Alternatively, the URL can be parsed and its hostname compared case-insensitively:

from urllib.parse import urlsplit


hostname = (
    urlsplit(link).hostname or ""
).lower()

if (
    hostname == "youtube.com"
    or hostname.endswith(".youtube.com")
):
    continue

The original link should still be retained unchanged for non-YouTube results.

Regression coverage should include:

• lowercase youtube.com;
• mixed-case YouTube.com;
• uppercase YOUTUBE.COM;
• common YouTube subdomains;
• normal non-YouTube URLs;
• preservation of retained URLs;
• interaction with max_results;
• malformed or missing links.

Targeted issue and pull-request searches found no existing report for this case-sensitive Google YouTube-filtering root.