Understanding Web Scraping Restrictions: Navigating Robots.txt Files and Website Policies for Successful Data Extraction