AWS SageMaker service init enumerates the entire (undeletable) training-job history unbounded — hangs/OOMs scans on accounts with large histories

Steps to Reproduce

Run a scan against an AWS account whose SageMaker training-job history is large. In our case (a real production account of an acquired company):

• 462,178 training jobs in one region (oldest 2018-06, newest 2021-05 — SageMaker offers no DeleteTrainingJob API, so this history is permanent by design)
• 143 models, 7 stopped notebook instances, 0 endpoints

Any of the following triggers the problem:

1. prowler aws --services sagemaker ...
2. prowler aws --check sagemaker_notebook_instance_encryption_enabled ... (or any other notebook/model/endpoint check — --check does not skip the service constructor)
3. a full scan that does not exclude sagemaker

Expected behavior

SageMaker checks complete in time proportional to the number of live resources (here: ~150), or at least the training-job enumeration is bounded so that checks unrelated to training jobs are unaffected.

Actual Result with Screenshots or Logs

SageMaker.__init__ (sagemaker_service.py) unconditionally:

1. paginates list_training_jobs over the entire account history with no time bound — measured at 20.4 minutes just to page through 462k summaries (~25k jobs/min, no throttling);
2. then schedules _describe_training_job per job and _list_tags_for_resource per job — ≈ 924,000 additional API calls for resources that last ran years ago.

Observed outcomes on a 6 GiB container (Kubernetes, prowler 5.39.0):

• a sagemaker-only run was OOM-killed after ~91 minutes;
• full-account runs including sagemaker hung for >12 hours without printing a single check header (the hang is inside service init, before any check runs);
• a run using --check with seven notebook/model/endpoint checks hit exactly the same hang, because the constructor does all listing regardless of which checks are selected.

The failure mode is nasty operationally: in a multi-account nightly batch the stuck service init consumed the whole Job deadline, and the reports of accounts that had already finished were withheld with it.

How did you install Prowler?

Docker (docker pull toniblyx/prowler)

Environment Resource

Kubernetes CronJob (EKS/GKE-style hub-and-spoke, --role assumed per account)

OS used

Container image toniblyx/prowler:5.39.0 (Python 3.12)

Prowler version

5.39.0 (current master shows the same constructor behaviour)

Pip version

n/a (official image)

Context

Two suggestions, either of which would fix us:

1. Bound the enumeration: pass CreationTimeAfter to list_training_jobs (and the analogous filters for processing/transform jobs), with a configurable lookback (e.g. aws.sagemaker_jobs_lookback_days, defaulting to something sane like 90–180 days). Note: use CreationTimeAfter, not LastModifiedTimeAfter — the latter has a known server-side pagination pathology that loops with empty pages until throttled (aws/aws-sdk#486).
2. Lazy-load per check group: only enumerate training/processing jobs when a check that needs them is selected. Today --check sagemaker_notebook_instance_* pays the full training-job cost for nothing.

Related prior art: #12502 is the same class of problem (per-resource describe fan-out making large-estate scans appear hung) on GCP BigQuery; #9608 addressed SageMaker tag-listing performance earlier.

Happy to provide more measurements if useful.