Let us help you find the training program you are looking for.

If you can't find what you are looking for, contact us, we'll help you find it. We have over 800 training programs to choose from.

Go Web Scraping

  • Course Code: Data Analysis / BI - Go Web Scraping
  • Course Dates: Contact us to schedule.
  • Course Category: Big Data & Data Science Duration: 2 Days Audience: This course is geared for Python experienced developers, analysts or others who wants to Learn how some Go-specific language features help to simplify building web scrapers along with common pitfalls and best practices regarding web scraping.

Course Snapshot 

  • Duration: 2 days 
  • Skill-level: Foundation-level Go Web Scraping skills for Intermediate skilled team members. This is not a basic class. 
  • Targeted Audience: This course is geared for Python experienced developers, analysts or others who wants to Learn how some Go-specific language features help to simplify building web scrapers along with common pitfalls and best practices regarding web scraping.  
  • Hands-on Learning: This course is approximately 50% hands-on lab to 50% lecture ratio, combining engaging lecture, demos, group activities and discussions with machine-based student labs and exercises. Student machines are required. 
  • Delivery Format: This course is available for onsite private classroom presentation, or remote instructor led delivery, or CBT/WBT (by request). 
  • Customizable: This course may be tailored to target your specific training skills objectives, tools of choice and learning goals. 

Web scraping is the process of extracting information from the web using various tools that perform scraping and crawling. Go is emerging as the language of choice for scraping using a variety of libraries. This course will quickly explain to you, how to scrape data data from various websites using Go libraries such as Colly and Goquery. The course starts with an introduction to the use cases of building a web scraper and the main features of the Go programming language, along with setting up a Go environment. It then moves on to HTTP requests and responses and talks about how Go handles them. You will also learn about a number of basic web scraping etiquettes. You will be taught how to navigate through a website, using a breadth-first and then a depth-first search, as well as find and follow links. You will get to know about the ways to track history in order to avoid loops and to protect your web scraper using proxies. Finally the course will cover the Go concurrency model, and how to run scrapers in parallel, along with large-scale distributed web scraping. 

Working in a hands-on learning environment, led by our Go Web Scraping expert instructor, students will learn about and explore: 

  • Use Go libraries like Goquery and Colly to scrape the web 
  • Common pitfalls and best practices to effectively scrape and crawl 
  • Learn how to scrape using the Go concurrency model 

Topics Covered: This is a high-level list of topics covered in this course. Please see the detailed Agenda below 

  • Implement Cache-Control to avoid unnecessary network calls 
  • Coordinate concurrent scrapers 
  • Design a custom, larger-scale scraping system 
  • Scrape basic HTML pages with Colly and JavaScript pages with chromedp 
  • Discover how to search using the “strings” and “regexp” packages 
  • Set up a Go development environment 
  • Retrieve information from an HTML document 
  • Protect your web scraper from being blocked by using proxies 
  • Control web browsers to scrape JavaScript sites 

Audience & Pre-Requisites 

This course is geared for attendees with Python skills who wish to Learn how some Go-specific language features help to simplify building web scrapers along with common pitfalls and best practices regarding web scraping. 

Pre-Requisites:  Students should have  

  • developers with some knowledge of Python.  

Course Agenda / Topics 

  1. Introducing Web Scraping and Go 
  • Introducing Web Scraping and Go 
  • What is web scraping? 
  • Why do you need a web scraper? 
  • What is Go? 
  • Why is Go a good fit for web scraping? 
  • How to set up a Go development environment 
  1. The Request/Response Cycle 
  • The Request/Response Cycle 
  • What do HTTP requests look like? 
  • What do HTTP responses look like? 
  • What are HTTP status codes? 
  • What do HTTP requests/responses look like in Go? 
  1. Web Scraping Etiquette 
  • Web Scraping Etiquette 
  • What is a robots.txt file? 
  • What is a User-Agent string? 
  • How to throttle your scraper 
  • How to use caching 
  1. Parsing HTML 
  • Parsing HTML 
  • What is the HTML format? 
  • Searching using the strings package 
  • Searching using the regexp package 
  • Searching using XPath queries 
  • Searching using Cascading Style Sheets selectors 
  1. Web Scraping Navigation 
  • Web Scraping Navigation 
  • Following links 
  • Submitting forms 
  • Avoiding loops 
  • Breadth-first versus depth-first crawling 
  • Navigating with JavaScript 
  1. Protecting Your Web Scraper 
  • Protecting Your Web Scraper 
  • Virtual private servers 
  • Proxies 
  • Virtual private networks 
  • Boundaries 
  1. Scraping with Concurrency 
  • Scraping with Concurrency 
  • What is concurrency 
  • Concurrency pitfalls 
  • The Go concurrency model 
  • sync package helpers 
  1. Scraping at 100x 
  • Scraping at 100x 
  • Components of a web scraping system 
  • Scraping HTML pages with colly 
  • Scraping JavaScript pages with chrome-protocol 
  • Distributed scraping with dataflowkit 
  • Summary 
View All Courses

    Course Inquiry

    Fill in the details below and we will get back to you as quickly as we can.

    Interested in any of these related courses?