Now this time we will learn How to Bulk Index URLs to the Search engine/ Google console for Indexing or Crawling purposes to help your Website SEO techniques.
You may see the need for the automation of this movement due to multiple reasons. For example, if the article is updated or article enhancement, etc. You usually try to index URLs to Google Search Engine using manual paths for your WordPress website or any other website.
Bulk Indexing will be a great SEO technique you can assume overall.
We shall cover the below factors in today’s post,
- Manual indexing and crawling problems
- Automatic indexing and crawling
- Submit multiple URLs to Google Search Engine
- Requirements for Automation – Request Permission
- Script for automation( Python,node.js, JAVA code, etc.)
Problems with Manual URL Indexing and Crawling
It is always hard to perform indexing manually like using the below search console window.
- To submit URLs, you must be log-in every time.
- Only authorized users like Admin only can index prosses after logging in.
- It takes much time, and if you have multiple URLs to send, you will have to submit them one by one manually.
- Crawling or indexing by Google can take some time ( wait may be more than days or weeks)
- It’s possible that you overlooked URLs to submit for index (including modified or newly added posts)
Automation for Url Indexing or crawling for Google
Using an automation process, all of the above-mentioned challenges can be fixed. We’ll create a script to do authentication and permission in this paragraph.
Once approved, the script can be edited with new Links or existing modified links to submit or resubmit as posts are updated or resubmitted.
With this method, you can submit multiple URLs to google as per the quota limitation.
The quota limit is 20000 URLs as default.
multiple Index URLs to Google Search Engine
We shall cover the below set of steps required to be executed,
- Create a Google Search console service account
- Site Ownership- Add your service account as a website owner
- with Written script and using an access token
- Submit URL Programmatically
A standard manual step involves below,
- Login to your Google Search Console account
- Submit each URL individually one by one
- and Wait for indexing to be effective
Before we begin to use the Indexing API, there is something that must be set up,
Authorize Requests to Indexing API
Application submissions must be permitted by an authenticated user who has access to that information.
- Request applications sent to the Indexing API must contain an authorization token.
- you have to Make sure to Enable Indexing API using the below this link,
Below we will cover high-level steps for performing the Authorization.
Create a Project
Before you can send submissions to the Indexing API, you need to inform Google about the customer and activate credentials to the API.
You do this by using the Google Console to create a new project or exiting project, which is a named group of settings and API access data, and register your application.
To get begun operating Indexing API, visit the setup tool, which instructs you through creating a new project in the Google API Console, enabling the API, and making credentials.
Create a service account
- Now you have to go Service accounts and Select your project else create a new project as explained above.
- Click add the Create Service Account option and enter the name like the project name for easy to identify, and give a short description. You can work with the default service account ID, or choose a different, memorable one. When you are done then click Create and continue button.
- The Service account permissions (optional) area that follows is not required. Click Continue.
- On the Grant users credentials to this service account. You can add a specific role or keep the role default owner.
- Once done you have to see your service account create option as below,
- In the main window panel, click on Action or: symbol. Please select the Manage keys option.
then select the format for your private key: JSON is recommended.
- Click on Create. A new public/private key pair will be generated and downloaded automatically to your machine;
Add your service account as a website owner
For site ownership, please complete the below steps,
Prove as the Owner of the website
The verification method of verifying that you own the property needs to be done by you.
Please note that verified owners will have access to sensitive Search data for a website.
Grant owner status to your service account
The next stage is to create your Service account (created in the above step) as a site owner.
This step is an important step toward fetching actual access to Indexing API.
- Please Open the Webmaster Console
- Click on the verified website.
- go to setting option
- In the Verified owner list option, click Add an owner.
- Provide your service account email as the delegated owner (before we created the service account email). You can locate your service account email address in two places:
client_email– This is an area in the JSON private key in the JSON service account file which we downloaded in one of the above stages.
An email has the below pattern,
<your-service-account>@<your-project-name>.google.com.iam.gserviceaccount.com
For example
hfdhj-api@gjjgj.iam.gserviceaccount.com
Final Step- Bulk Submit URLs Programmatically
That’s all, you are now all ready to set up the automation. With needed requirements to start gathering the Google Indexing API.
As a next step, please use the scripts of Python (we are comfortable with Python) to call the Google Indexing API.
Please see the below link for the final stages where we will use a Python script to send the URL for Google indexing programmatically.
Please Download the Zip file to get the script for the indexing or crawling sense using automation processes (Now this time we have only one script).
The next step is to open your terminal and run these commands..
Windows user
1. pip install pip
2. pip install oauth2client httplib2
3. pip install pandas
Mac user
Code 1: pip3 install oauth2client httplib2
Code 2: pip3 install pandasclose your current terminal and go to the Folder that you have downloaded before, then unZip that file (IndexAPI). run command python indexing.py
all done, this script will do automation process for your URL indexing or crawling to search engine.
Do you have any statements or ideas or any helpful suggestions to share?
Please write your comments below.
