A script for scraping all the EPG sources.

[Stealthx3]

This is an .sh script that serves as the trigger-pull for the IPTV-EPG project currently on my system. It was created to solve problems encountered when attempting to scrape all possible EPG sources.

update-epg.sh

Prerequisites:

• To use this you will need to create an public/guides directory within your epg directory created by the git clone.
• Set the PARALLEL, NODE_HEAP_MB, and GRAB_DAYS environment variables prior to executing the script.
• The script expects to exist in the same directory as your EPG folder.

The command used to trigger the script should look something like this:
PARALLEL=4 NODE_HEAP_MB=3072 GRAB_DAYS=3 epg/../update-epg.sh

Problems this script attempts to address:

• Controls memory heap size and the resulting crashes by writing data collected to individualized XML files on a per-site basis, as opposed to after all EPG sources are scraped.
• Provides the option to run a set number of per-site scrapes in parallel, and to fine tune the amount of HEAP memory each instance can utilize to prevent more significant system-wide OOM errors.
• Automatically extracts site URLs from the sites.md file to a list.txt file for , and randomizes the order to attempt to try and avoid triggering DDOS issues down a particular pipe.

Its function is currently divided into 3 phases:
Phase 1:

• The script uses git to update all relevant files in the project.
• The script isolates, and extracts all IPTV urls from the SITES.md file and place them in a separate list.txt file for the script to work through in phase 2.
• The script randomizes the order of urls in the new list.txt file

Phase 2:
The script reads each individual url in list.txt and runs them in place of "example.com" in the following command, at a time, until the list is completed:
npm run grab --- --sites=example.com --output=public/guides/{site}.xml

Phase 3:
The script will copies the contents of all the individual xml files in the public/guides directory into one monolithic xml file and ensures it properly reads out as an xmltv file.

Afterword:
A better way to handle this long term on the project side would probably be to break up the individual channel retrievals into more abstract blocks of a size that results in less than a set amount of heap allocated, as that would also help mitigate bans triggered by DDOS detection at each website and allow for more simultaneous connections since the individual connections they are more spread out.

Yes, I made Claude do it but I made sure to audit the results and now you can too.