Golang uses selenium to operate Chrome
1. Demand
Solve the problem of automatic login, and solve the problem of crawler by the way.
2. Basic concepts
Selenium: selenium is a tool for Web application testing. Selenium testing runs directly and automatically in the browser, just like real users operate manually.
Webdriver: Chrome driver is an automated testing tool provided by Google for website developers.
Selenium and webdriver were originally two different open source projects. Later, selenium 2 merged selenium 1 (RC) and webdriver, using the name of selenium, but the implementation method and protocol basically followed webdriver. It can be seen as the same.
In short, you need to call chrome through the chrome driver to simulate browser operation.
3. Installation
-
Download the chrome driver. The version of chrome and chrome driver should correspond, Chrome driver version download , put it in the corresponding directory.
-
Download the golang code package. The golang code package of selenium
go get -t -d github.com/tebeka/selenium
- Note: my Win10 calls chrome to report an error. It mainly involves three points: 1. Add chrome to the environment variable path, which can be run directly through cmd chrome.exe Determine whether the program runs successfully; 2. Modify the permissions of the program to let the executing account obtain the ownership, and avoid the elevation of permissions; 3. If you are prompted that Google Chrome is not found, copy a copy chrome.exe Rename to google.chrome.exe .
4. Source code analysis
4.1 selenium execution process analysis
After my understanding and thinking, I think the main operation mode of selenium is as follows (personal understanding, for reference only).
The program needs to call selenium library to execute corresponding functions, call chrome browser in the background, and then drive the request of operation elements to the browser below. The browser driver forwards the request to the browser. Finally, the result is returned.
4.2 source file analysis selenium-golangdoc
Because the selenium source package is not very large, and because it's chrome for actual combat, I cut the source code and add comments. The godoc package should not be completely written. For example, webdriver and webelement only talk about the interface, but not the implementation details. We can brain supplement according to selenium operation. You can refer to the implementation of selenium in python, such as automated testing .
Let's continue to draw a picture to explain the package.
Brief analysis of source code:
// Category I miscellaneous
//Delete session
func DeleteSession(urlPrefix, id string) error
//Turn debug ging on and off
func SetDebug(debug bool)
//Set up proxy
func (c Capabilities) AddProxy(p Proxy)
//Set log level
func (c Capabilities) SetLogLevel(typ log.Type, level log.Level)
// The second type of seleium background service
//Optional service instance
type ServiceOption func(*Service) error
//Add chrome path information, return serviceOption type
func ChromeDriver(path string) ServiceOption
//Structure of the service, including hidden types
type Service struct {
// contains filtered or unexported fields
}
//The server that starts the chrome browser and returns the service type pointer
func NewChromeDriverService(path string, port int, opts ...ServiceOption) (*Service, error)
//Shut down the service. Remember to shut down defer
func (s *Service) Stop() error
// The third kind of chrome operation related
//Set browser compatibility, map type, such as chrome browser compatibility.
//caps := selenium.Capabilities{"browserName": "chrome"}
type Capabilities map[string]interface{}
//Add chrome compatibility by calling functions
func (c Capabilities) AddChrome(f chrome.Capabilities)
//Start webdriver instance
func NewRemote(capabilities Capabilities, urlPrefix string) (WebDriver, error)
//Through the WebDriver interface, we can see that the implementation method of the specific page is the interface, and the implementation method is in the interface.
type WebDriver interface {
//Returns version information for the server environment
// Status returns various pieces of information about the server environment.
Status() (*Status, error)
//Create a new session
// NewSession starts a new session and returns the session ID.
NewSession() (string, error)
//Create a new session (obsolete)
// SessionId returns the current session ID
// Deprecated: This identifier is not Go-style correct. Use SessionID
// instead.
SessionId() string
//Get a new ssionid
// SessionID returns the current session ID.
SessionID() string
//Switch session
// SwitchSession switches to the given session ID.
SwitchSession(sessionID string) error
//Return to compatibility
// Capabilities returns the current session's capabilities.
Capabilities() (Capabilities, error)
//Set asynchronous script execution time
// SetAsyncScriptTimeout sets the amount of time that asynchronous scripts
// are permitted to run before they are aborted. The timeout will be rounded
// to nearest millisecond.
SetAsyncScriptTimeout(timeout time.Duration) error
//Set the time to wait for the search element. Purpose: if the page result returns slowly, you need to wait for the page content to return completely, and then operate the page element.
// SetImplicitWaitTimeout sets the amount of time the driver should wait when
// searching for elements. The timeout will be rounded to nearest millisecond.
SetImplicitWaitTimeout(timeout time.Duration) error
//Set the time to wait for the page
// SetPageLoadTimeout sets the amount of time the driver should wait when
// loading a page. The timeout will be rounded to nearest millisecond.
SetPageLoadTimeout(timeout time.Duration) error
//Set session exit
// Quit ends the current session. The browser instance will be closed.
Quit() error
//Get the current window handle, a sequence number, open a window and a handle
// CurrentWindowHandle returns the ID of current window handle.
CurrentWindowHandle() (string, error)
//Get all open window handles, get all window handles
// WindowHandles returns the IDs of current open windows.
WindowHandles() ([]string, error)
//Return the URL of the current page connection
// CurrentURL returns the browser's current URL.
CurrentURL() (string, error)
//Get the title of the current page
// Title returns the current page's title.
Title() (string, error)
//Returns all contents of the current page
// PageSource returns the current page's source.
PageSource() (string, error)
//Close the current window
// Close closes the current window.
Close() error
//Switch the frame. A complete html is embedded in the frame. If the content in the operation needs to enter iframe. Switch frame (NIL), back to the top
// SwitchFrame switches to the given frame. The frame parameter can be the
// frame's ID as a string, its WebElement instance as returned by
// GetElement, or nil to switch to the current top-level browsing context.
SwitchFrame(frame interface{}) error
//Switch windows to the specified window
// SwitchWindow switches the context to the specified window.
SwitchWindow(name string) error
//close window
// CloseWindow closes the specified window.
CloseWindow(name string) error
//Set maximize window
// MaximizeWindow maximizes a window. If the name is empty, the current
// window will be maximized.
MaximizeWindow(name string) error
//Set window size
// ResizeWindow changes the dimensions of a window. If the name is empty, the
// current window will be maximized.
ResizeWindow(name string, width, height int) error
//Navigate to the corresponding interface through the url. The main option is to open the url address.
// Get navigates the browser to the provided URL.
Get(url string) error
//Roll forward
// Forward moves forward in history.
Forward() error
//Backward flip
// Back moves backward in history.
Back() error
//Refresh
// Refresh refreshes the page.
Refresh() error
//Find and locate an html element.
// FindElement finds exactly one element in the current page's DOM.
FindElement(by, value string) (WebElement, error)
//Find and locate multiple html elements
// FindElement finds potentially many elements in the current page's DOM.
FindElements(by, value string) ([]WebElement, error)
//Get current focus element
// ActiveElement returns the currently active element on the page.
ActiveElement() (WebElement, error)
//Decode element response
// DecodeElement decodes a single element response.
DecodeElement([]byte) (WebElement, error)
//Decoding multiple element responses
// DecodeElements decodes a multi-element response.
DecodeElements([]byte) ([]WebElement, error)
//Get all cookie s
// GetCookies returns all of the cookies in the browser's jar.
GetCookies() ([]Cookie, error)
//Get the specified cookie
// GetCookie returns the named cookie in the jar, if present. This method is
// only implemented for Firefox.
GetCookie(name string) (Cookie, error)
//Add cookie to jar
// AddCookie adds a cookie to the browser's jar.
AddCookie(cookie *Cookie) error
//Delete all cookie s
// DeleteAllCookies deletes all of the cookies in the browser's jar.
DeleteAllCookies() error
//Delete the specified cookie
// DeleteCookie deletes a cookie to the browser's jar.
DeleteCookie(name string) error
//Tap the mouse button
// Click clicks a mouse button. The button should be one of RightButton,
// MiddleButton or LeftButton.
Click(button int) error
//Double click the mouse button
// DoubleClick clicks the left mouse button twice.
DoubleClick() error
//Mouse down
// ButtonDown causes the left mouse button to be held down.
ButtonDown() error
//Raise the mouse
// ButtonUp causes the left mouse button to be released.
ButtonUp() error
//Send changes to active element (discarded)
// SendModifier sends the modifier key to the active element. The modifier
// can be one of ShiftKey, ControlKey, AltKey, MetaKey.
//
// Deprecated: Use KeyDown or KeyUp instead.
SendModifier(modifier string, isDown bool) error
//Send key sequence sequence to active element
// KeyDown sends a sequence of keystrokes to the active element. This method
// is similar to SendKeys but without the implicit termination. Modifiers are
// not released at the end of each call.
KeyDown(keys string) error
//Release sent elements
// KeyUp indicates that a previous keystroke sent by KeyDown should be
// release
KeyUp(keys string) error
//take snapshot
// Screenshot takes a screenshot of the browser window.
Screenshot() ([]byte, error)
//Log grabbing
// Log fetches the logs. Log types must be previously configured in the
// capabilities.
//
// NOTE: will return an error (not implemented) on IE11 or Edge drivers.
Log(typ log.Type) ([]log.Message, error)
//Disarm alarm
// DismissAlert dismisses current alert.
DismissAlert() error
//Accept alert
// AcceptAlert accepts the current alert.
AcceptAlert() error
//Return to current alert content
// AlertText returns the current alert text.
AlertText() (string, error)
//Send alert content
// SetAlertText sets the current alert text.
SetAlertText(text string) error
//Execute script
// ExecuteScript executes a script.
ExecuteScript(script string, args []interface{}) (interface{}, error)
//Execute script asynchronously
// ExecuteScriptAsync asynchronously executes a script.
ExecuteScriptAsync(script string, args []interface{}) (interface{}, error)
//Execute source script
// ExecuteScriptRaw executes a script but does not perform JSON decoding.
ExecuteScriptRaw(script string, args []interface{}) ([]byte, error)
//Execute source script asynchronously
// ExecuteScriptAsyncRaw asynchronously executes a script but does not
// perform JSON decoding.
ExecuteScriptAsyncRaw(script string, args []interface{}) ([]byte, error)
//Waiting condition is true
// WaitWithTimeoutAndInterval waits for the condition to evaluate to true.
WaitWithTimeoutAndInterval(condition Condition, timeout, interval time.Duration) error
//waiting time
// WaitWithTimeout works like WaitWithTimeoutAndInterval, but with default polling interval.
WaitWithTimeout(condition Condition, timeout time.Duration) error
//wait for
//Wait works like WaitWithTimeoutAndInterval, but using the default timeout and polling interval.
Wait(condition Condition) error
}
//Subsequent implementation of related elements, interface type, including implementation methods
type WebElement interface {
// click the selected element
// Click clicks on the element.
Click() error
//Send data to selected elements
// SendKeys types into the element.
SendKeys(keys string) error
//Submit button
// Submit submits the button.
Submit() error
//Clear button
// Clear clears the element.
Clear() error
//Move elements to corresponding coordinates
// MoveTo moves the mouse to relative coordinates from center of element, If
// the element is not visible, it will be scrolled into view.
MoveTo(xOffset, yOffset int) error
// Find child elements
// FindElement finds a child element.
FindElement(by, value string) (WebElement, error)
//Find multiple child elements
// FindElement finds multiple children elements.
FindElements(by, value string) ([]WebElement, error)
//Return label name
// TagName returns the element's name.
TagName() (string, error)
//Return element content
// Text returns the text of the element.
Text() (string, error)
//Element selected returns true
// IsSelected returns true if element is selected.
IsSelected() (bool, error)
//Return true if element enabled
// IsEnabled returns true if the element is enabled.
IsEnabled() (bool, error)
//Return true if element display
// IsDisplayed returns true if the element is displayed.
IsDisplayed() (bool, error)
//Get the name of the element
// GetAttribute returns the named attribute of the element.
GetAttribute(name string) (string, error)
//Location of range element
// Location returns the element's location.
Location() (*Point, error)
//Returns the position of an element after scrolling
// LocationInView returns the element's location once it has been scrolled
// into view.
LocationInView() (*Point, error)
//Returns the size of the element
// Size returns the element's size.
Size() (*Size, error)
//Return css priority
// CSSProperty returns the value of the specified CSS property of the
// element.
CSSProperty(name string) (string, error)
//Returns a snapshot of property scrolling
// Screenshot takes a screenshot of the attribute scroll'ing if necessary.
Screenshot(scroll bool) ([]byte, error)
}
4.3 description
-
You may need to understand the basic concepts of Html, Css and JavaScript. Rookie tutorial
-
You may need to understand the Dom structure. html Dom
-
You may need to know XPATH,CSSSelector. Details of using Xpath and CSS selectors
-
You can refer to the selenium operation implementation in python. After all, there are many python cases. selenim operation daily record
- Refer to learn the automation course of white moon and black feather, b station video synchronization, push. automated testing
5. Basic operation
Before operation, share a quick method to get CSS Selector and xpath.
You can use chrome browser to visit the web page, press F12, click the Elements arrow in the upper left corner of the debugging page, and then move the mouse to the destination to display the HTML Elements corresponding to the page.
Right click the selected element and select copy. At this time, you can select the address of ID,CLASS,CSS Selector or Xpath according to the direct selection
Refer to other people's cases and write a few small cases for reference only. Light up the code.
-
Open Baidu and search automatically.
package main import ( "fmt" "github.com/tebeka/selenium" "time" ) const ( //Set constants separately chromedriver.exe Address and local call port of seleniumPath = `H:\webdriver\chromedriver.exe` port = 9515 ) func main() { //1. Enable selenium service //Set the option of the selium service to null. Set as needed. ops := []selenium.ServiceOption{} service, err := selenium.NewChromeDriverService(seleniumPath, port, ops...) if err != nil { fmt.Printf("Error starting the ChromeDriver server: %v", err) } //Delay service shutdown defer service.Stop() //2. Call browser //Set browser compatibility. We set the browser name to chrome caps := selenium.Capabilities{ "browserName": "chrome", } //Call browser urlPrefix: test reference: defaulturlprefix =“ http://127.0.0.1 :4444/wd/hub" wd, err := selenium.NewRemote(caps, "http://127.0.0.1:9515/wd/hub") if err != nil { panic(err) } //Delay exiting chrome defer wd.Quit() //3. Operate on page elements //Get Baidu page if err := wd.Get("https://www.baidu.com/"); err != nil { panic(err) } //Find Baidu input box id we, err := wd.FindElement(selenium.ByID, "kw") if err != nil { panic(err) } //Send '' to input box err = we.SendKeys("the best in all the land") if err != nil { panic(err) } //Find Baidu submit button id we, err = wd.FindElement(selenium.ByID, "su") if err != nil { panic(err) } //Click Submit err = we.Click() if err != nil { panic(err) } //Quit after 20 seconds of sleep time.Sleep(20 * time.Second) } -
Embedded iframe switch.
package main import ( "fmt" "github.com/tebeka/selenium" "time" ) const ( //Set constants separately chromedriver.exe Address and local call port of seleniumPath = `H:\webdriver\chromedriver.exe` port = 9515 ) func main() { //1. Enable selenium service //Set the option of the selium service to null. Set as needed. ops := []selenium.ServiceOption{} service, err := selenium.NewChromeDriverService(seleniumPath, port, ops...) if err != nil { fmt.Printf("Error starting the ChromeDriver server: %v", err) } //Delay service shutdown defer service.Stop() //2. Call browser //Set browser compatibility. We set the browser name to chrome caps := selenium.Capabilities{ "browserName": "chrome", } //Call browser urlPrefix: test reference: defaulturlprefix =“ http://127.0.0.1 :4444/wd/hub" wd, err := selenium.NewRemote(caps, "http://127.0.0.1:9515/wd/hub") if err != nil { panic(err) } //Delay exiting chrome defer wd.Quit() //3. operate on page elements //Get test page if err := wd.Get("http://cdn1.python3.vip/files/selenium/sample2.html"); err != nil { panic(err) } //4. Switch to the corresponding frame //wd.SwitchFrame (webelement obtained by id or frame). We use two methods to implement it respectively. //4.1 find id=frame1 by frame ID /* err = wd.SwitchFrame("frame1") if err != nil { panic(err) } // At this time, navigate to the html of iframe, and then use bycssselector // Because animal contains multiple objects, we use findelements wes, err := wd.FindElements(selenium.ByCSSSelector, ".animal") if err != nil { panic(err) } //Loop to get information for each element for _,we := range wes { text, err := we.Text() if err != nil { panic(err) } fmt.Println(text) } */ //4.2 the webelement obtained by frame is realized by switching webelement. // Find ifname's webelement object element, err := wd.FindElement(selenium.ByCSSSelector, "#frame1") // Different ways to get element //element, err := wd.FindElement(selenium.ByCSSSelector, `iframe[name="innerFrame"]`) if err != nil { panic(err) } //Switch to iframe err = wd.SwitchFrame(element) if err != nil { panic(err) } // At this time, navigate to the html of iframe, and then use bycssselector // Because animal contains multiple objects, we use findelements wes, err := wd.FindElements(selenium.ByCSSSelector, ".animal") if err != nil { panic(err) } //Loop to get information for each element for _, we := range wes { text, err := we.Text() if err != nil { panic(err) } fmt.Println(text) } //5. Switch back to the top-level frame, because the outer value element cannot be operated in the frame, so we need to switch it out //frame=nil is to switch back to the top frame err = wd.SwitchFrame(nil) if err != nil { panic(err) } //Select elements according to class name we, err := wd.FindElement(selenium.ByCSSSelector, ".baiyueheiyu") if err != nil { panic(err) } //View the contents of the top-level element fmt.Println(we.Text()) //Quit after 20 seconds of sleep time.Sleep(20 * time.Second) } -
Multiple windows switching.
package main import ( "fmt" "github.com/tebeka/selenium" "strings" "time" ) const ( //Set constants separately chromedriver.exe Address and local call port of seleniumPath = `H:\webdriver\chromedriver.exe` port = 9515 ) func main() { //1. Enable selenium service //Set the option of selenium service to null. Set as needed. ops := []selenium.ServiceOption{} service, err := selenium.NewChromeDriverService(seleniumPath, port, ops...) if err != nil { fmt.Printf("Error starting the ChromeDriver server: %v", err) } //Delay service shutdown defer service.Stop() //2. Call browser instance //Set browser compatibility. We set the browser name to chrome caps := selenium.Capabilities{ "browserName": "chrome", } //Call browser urlPrefix: test reference: defaulturlprefix =“ http://127.0.0.1 :4444/wd/hub" wd, err := selenium.NewRemote(caps, "http://127.0.0.1:9515/wd/hub") if err != nil { panic(err) } //Delay exiting chrome defer wd.Quit() //3. Open multi page chrome instance //Now I think of two ways to open it, //The first is that there is a url connection in the page, which can be opened by click ing () //The second way is to open it by script. wd.ExecuteScript if err := wd.Get("http://cdn1.python3.vip/files/selenium/sample3.html"); err != nil { panic(err) } //The first way is to find the url address in the page and jump the page we, err := wd.FindElement(selenium.ByTagName, "a") if err != nil { panic(err) } we.Click() //The second way is to open a new window by running the general js script, because we don't need to get the result of the operation temporarily, and we don't get the return value. wd.ExecuteScript(`window.open("https://www.qq.com", "_blank");`, nil) wd.ExecuteScript(`window.open("https://www.runoob.com/jsref/obj-window.html", "_blank");`, nil) //This line is to send alarm information. The purpose of writing this line is to see which main window is currently wd.ExecuteScript(`window.alert(location.href);`, nil) //View the handle value of the current window handle, err := wd.CurrentWindowHandle() if err != nil { panic(err) } fmt.Println(handle) fmt.Println("--------------------------") //View handle values for all pages handles, err := wd.WindowHandles() if err != nil { panic(err) } for _, handle := range handles { fmt.Println(handle) } fmt.Println("--------------------------") //4. Jump to the specified web page //Although we have opened multiple pages, our current handle value is still the first page. We need to find a way to handle it. //Remember to save the handle value of the current main page //mainhandle := handle //Make the corresponding webpage by judging the conditions //Get all handle values handles, err = wd.WindowHandles() if err != nil { panic(err) } //Traverse all the handle values, find the target page through the url, judge the equality, break out, and stop at the corresponding page. for _, handle := range handles { wd.SwitchWindow(handle) url, _ := wd.CurrentURL() if strings.Contains(url, "qq.com") { break } } //View the handle of this page handle, err = wd.CurrentWindowHandle() if err != nil { panic(err) } fmt.Println(handle) //This line is to send alarm information. The purpose of writing this line is to see which main window is currently wd.ExecuteScript(`window.alert(location.href);`, nil) //Switch back to the first page //wd.SwitchWindow(mainhandle) //Quit after 20 seconds of sleep time.Sleep(20 * time.Second) } -
Single selection, multi selection box operation.
package main import ( "fmt" "github.com/tebeka/selenium" "time" ) const ( //Set constants separately chromedriver.exe Address and local call port of seleniumPath = `H:\webdriver\chromedriver.exe` port = 9515 ) func main() { //1. Enable selenium service //Set the option of selenium service to null. Set as needed. ops := []selenium.ServiceOption{} service, err := selenium.NewChromeDriverService(seleniumPath, port, ops...) if err != nil { fmt.Printf("Error starting the ChromeDriver server: %v", err) } //Delay service shutdown defer service.Stop() //2. Call browser instance //Set browser compatibility. We set the browser name to chrome caps := selenium.Capabilities{ "browserName": "chrome", } //Call browser urlPrefix: test reference: defaulturlprefix =“ http://127.0.0.1 :4444/wd/hub" wd, err := selenium.NewRemote(caps, "http://127.0.0.1:9515/wd/hub") if err != nil { panic(err) } //Delay exiting chrome defer wd.Quit() // 3. radio, checkbox and select box operation (functions to be improved, https://github.com/tebeka/selenium/issues/141) if err := wd.Get("http://cdn1.python3.vip/files/selenium/test2.html"); err != nil { panic(err) } //3.1 radio operation we, err := wd.FindElement(selenium.ByCSSSelector, `#s_radio > input[type=radio]:nth-child(3)`) if err != nil { panic(err) } we.Click() //3.2 operating multiple checkbox es //Delete default checkbox we, err = wd.FindElement(selenium.ByCSSSelector, `#s_checkbox > input[type=checkbox]:nth-child(5)`) if err != nil { panic(err) } we.Click() //Select options we, err = wd.FindElement(selenium.ByCSSSelector, `#s_checkbox > input[type=checkbox]:nth-child(1)`) if err != nil { panic(err) } we.Click() we, err = wd.FindElement(selenium.ByCSSSelector, `#s_checkbox > input[type=checkbox]:nth-child(3)`) if err != nil { panic(err) } we.Click() //3.3 select multiple //Remove default options //Select Default we, err = wd.FindElement(selenium.ByCSSSelector, `#ss_multi > option:nth-child(3)`) if err != nil { panic(err) } we.Click() we, err = wd.FindElement(selenium.ByCSSSelector, `#ss_multi > option:nth-child(2)`) if err != nil { panic(err) } we.Click() //Quit after 20 seconds of sleep time.Sleep(20 * time.Second) }
Conclusion
Personal understanding, for reference only. If there is any mistake, please correct it.
Always on the road, quietly forward.