Search Engine Blogs

bingbot Series: Get your content indexed fast by now submitting up to 10,000 URLs per day to Bing

Bing's Webmaster Blog -

Today, we are excited to announce a significant increase in the number of URLs webmasters can submit to Bing to get their content crawled and indexed immediately. We believe that enabling this change will trigger a fundamental shift in the way that search engines, such as Bing, retreive and are notified of new and updated content across the web. Instead of Bing monitoring often RSS and similar feeds or frequently crawling websites to check for new pages, discover content changes and/or new outbound links, websites will notify the Bing directly about relevant URLs changing on their website. This means that eventually search engines can reduce crawling frequency of sites to detect changes and refresh the indexed content.  For many years, Bing has offered all webmasters the ability to submit their site URLs through the Bing Webmaster Tools portal as well as the Bing Webmaster Tools API for immediate crawl and indexation. Until today, this feature was throttled for all sites to submit maximum of 10 URLs per day and maximum of 50 URLs per month.  Today we are releasing the Adaptive URL submission feature that increases the daily quota by 1000x, allowing you to submit up to 10,000 URLs per day, with no monthly quotas. The daily quota per site will be determined based on the site verified age in Bing Webmaster tool, site impressions and other signals that are available to Bing. Today the logic is as follows, and we will tweak this logic as needed based on usage and behavior we observe.    Key things to note: Site verified age is one of the signals but not the only signal that is used to determine the daily URL quota per site. Webmasters should be able to see revised limit for their site on Bing webmaster tools portal (Submit URLs option) or by using the Get URL submission quota API.  Webmaster tools portal ​ Get URL submission quota API In Bing webmaster tools portal, under “Submit URLs” option, webmaster will see maximum of 1,000 latest submitted URLs even though the permissible quota per day could be greater than 1,000 As per existing functionality, for sub-domain level site “Submit URL” option will not be applicable. If there are sites mapped to sub-domain then quota is allocated at domain level and not at sub-domain level. So, login to Bing Webmaster Tools or integrate Bing Webmaster APIs in your Content Management Systems now to benefit from the increase and contact us for feedback.  Feel free to contact us also if your web site requires more than 10,000 URLs submitted per day. We will adjust as needed.   Thanks! Bing Webmaster Tools team  

Introducing Clarity, a web analytics product for webmasters

Bing's Webmaster Blog -

Today, we are announcing the beta release of Clarity, a web analytics product which enables website developers to understand user behavior at scale.  Web developers face many challenges in building compelling and user-friendly websites. Understanding why users struggle, where users run into issues or why they abandon a website is difficult. When making updates to a web experience, A/B experimentation helps developers decide on which way to go. While A/B experiments allow developers to see when their key metrics are moving, the primary drawback is the lack of visibility into why the metrics moved in any given direction. This gap in understanding user behavior led us to build Clarity. Session Replay The session replay capability of Clarity allows web developers to view a user's page impression to understand user interactions such as mouse movement, touch gestures, click events and much more. Being able to replay user sessions allows web developers to empathize with users and understand their pain points. Clarity Case Studies  Bing uses Clarity to detect poor user experience due to malware In the Wild West of devices and browsers, the experience you think you are serving and what your users see can be completely different - devices, plugin, proxies and networks can all change and degrade the quality of your experience. These problems are expensive to diagnose and fix.   The first image shows the page of Bing with malware while the second image shows Bing default  experience after removal of malware.  Clarity has been used by the Bing UX team at Microsoft to delve into sessions that had negative customer satisfaction and determine what went wrong. In some cases, engineers were able to detect pages that had multiple overlays of advertisements and looked nothing like the expected experience for customers. Looking through the layout anomalies and network request logs that Clarity provides, Bing developers were able to diagnose the cause: malware installed on the end user's machine was hijacking the page and inserting bad content.  With the knowledge of what was causing these negative experiences, Bing engineers were able to design and implement changes which defended the Bing page. By doing so they increased revenue while decreasing page load time - all while giving their customers a significantly improved experience.  Cook with Manali uses Clarity to improve user engagement  Cook with Manali is a food blog and like many other blogs dedicated to cooking, posts begin with a story about the inspiration behind the recipe. Posts have detailed instructions to prepare the meal, high-quality photographs of the finished dish and potentially step by step pictures to help explain the more complicated parts. Near the bottom of the page is a shorthand recipe card summarizing ingredients, instructions and nutritional information. While this long post format enables food bloggers to emotionally connect with their readers and preemptively address any complication in the recipe, some readers would rather get straight to the recipe.  When the Cook with Manali team started using Clarity, they were able to investigate real user sessions and realized that almost thirty percent of users were abandoning the page before reaching the bottom of these recipe pages, which has important information about the recipe. In many cases, it seemed that users felt they had to scroll too far to get to the recipe that they really cared about and lost patience before making it far enough on the page. The developers realized their strategy was backfiring and creating a bad experience for some of their users, prompting them to add a "Jump to Recipe" button at the top of these pages.   With the new button deployed, the team was able to see traffic going up and abandonment going down. When they dug into these new session replays, they were able to see users utilizing the new button and getting directly to the content they cared about. They saw abandonment for these pages drop down to roughly ten percent, signaling a significant increase in user satisfaction. Interestingly, many users now utilize the "Jump to Recipe" button to then scroll back up to read the larger story afterwards.   How does Clarity work?  Clarity works on any HTML webpage (desktop or mobile) after adding a small piece of JavaScript to the website. This JavaScript code listens to browser events and instruments layout changes, network requests and user interactions. The instrumentation data is then uploaded and stored in the Clarity server running on Microsoft Azure.   Other capabilities coming soon  Interesting sessions are automatically bubbled up based on Clarity's AI and machine learning capabilities to help web developers review user sessions with abnormal click or scroll behavior, session length, JavaScript errors and more. Web developers can spend less time and gain more insight into their users focusing on the sessions that Clarity marks as most relevant.  Related sessions are a grouping of similar sessions that are recommended based a single session. This feature allows web developers to quickly understand the scope of a specific user behavior and find other occurrences for the same user as well as other users.   Heatmaps provide a view into user behavior at an aggregate level through click/touch and scroll heatmaps. Click/touch heatmap provides distribution of user interactions across a webpage. Scroll heatmaps provide how far users scroll on your webpage.   How do I get started?  Sign up at the Clarity website using your Microsoft Account! (In case you don’t have one, you can sign-up here.)   When you create a new project, it will be added to the waitlist. A notification will be sent when your project is approved for onboarding and you can login to Clarity to retrieve the uniquely generated JS code for your project. Once you have added the code to your website, you can use Clarity dashboard to start replaying user sessions and gain insights.   Please reach out to ClarityMS@microsoft.com if you have any questions.   Contributing to Clarity  The Clarity team has also open sourced the JavaScript library which instruments pages to help understand user behavior on websites on GitHub . As Clarity is in active development with continuous improvements, join our community and contribute. Getting started is easy, just visit GitHub and read through our README.    Here are some of the exciting new feature the Clarity team is brewing up: Interesting sessions are automatically bubbled up based on Clarity's AI and machine learning capabilities to help web developers review user sessions with abnormal click or scroll behavior, session length, JavaScript errors and more. Web developers can spend less time and gain more insight into their users focusing on the sessions that Clarity marks as most relevant. Related sessions are a grouping of similar sessions that are recommended based a single session. This feature allows web developers to quickly understand the scope of a specific user behavior and find other occurrences for the same user as well as other users. Heatmaps provide a view into user behavior at an aggregate level through click/touch and scroll heatmaps. Click/touch heatmap provides distribution of user interactions across a webpage. Scroll heatmaps provide how far users scroll on your webpage. Thank you,

bingbot Series: Getting most of Bingbot via Bing Webmaster Tools

Bing's Webmaster Blog -

There are multiple features in Bing Webmaster Tools that allows webmasters to check Bingbot’s performance and issues on their site, provide input to Bingbot crawl schedules and check if that random bot hitting the pages frequently is actually Bingbot or not.  In part 4 of our Bingbot series, Nikunj Daga, Program Manager for Bing Webmaster Tools, revisits some of those tools and features that assists webmasters in troubleshooting and optimizing Bingbot’s performance on their site.  Crawl Information - Webmasters can get the data about Bingbot’s performance on their site in the Reports and Data section in Bing Webmaster Tools. The site activity chart present in this page can show an overlapping view of total pages indexed, pages crawled and pages where there were crawl errors for the last six months along with the Impressions and Clicks data. Through this chart, it is easier for webmasters to visually see whether the changes they made on their sites had any impact on page crawling.   Further, in order to get more information on the pages with crawl errors, webmasters can go to the Crawl Information page. In this page, an aggregated count of the pages with different errors that Bingbot faced is provided along with the list of those URLs. This make it simple for webmasters to troubleshoot why a particular page that they are looking for in Bing is not appearing while searching.  Crawl Errors – In addition to webmasters going and checking the Crawl Information page for crawl errors, Bing Webmaster Tools also proactively notifies the webmasters in case Bingbot faces significant number of issues while crawling the site. These notifications are sent on the Message Center in Bing Webmaster Tools. These alerts can also be sent through email to users who do not visit webmaster tools on a regular basis. So, it will not be a bad idea for webmasters to opt in for the email communication from Bing Webmaster Tools through the Profile Page.  Webmasters can set the preference for kind of alerts they want to receive emails for along with the preferred contact frequency.  Further, Bingbot can face different kinds of errors while crawling a site, a detailed list of which along with their description and action can be found here.  Crawl Control – The Crawl Control feature allows webmasters to provide input to the Bingbot about the crawl speed and timing for your site. It can be found under the “Configure My Site” section in Bing Webmaster Tools. Using this feature, you can set hourly crawl rate for your site and notify Bingbot to crawl slowly during peak business hours and faster during off peak hours. There are preset schedules to choose from based on the most common business hours followed across the globe. In addition to the preset schedules, webmasters also have the option to fully customize the crawl schedule based on their site’s traffic pattern. Customizing the crawl pattern is very easy and can be done by just dragging and clicking on the graph present in the Crawl Control feature.  Fetch as Bingbot – The Fetch as Bingbot tool returns the code that Bingbot sees when it crawls the page. Webmasters can find this feature under the “Diagnostics and Tools” section to submit a request to fetch as Bingbot. Once the fetch is completed, the status will change from “Pending” to “Completed” and the webmasters will be able to see the code that appears to Bingbot when it tries to crawl the site. This is a useful feature for webmasters who use dynamic content on their sites and is the basic check if they want to see what data Bingbot sees among all the dynamic and static content on the site.  Verify Bingbot - Found under the “Diagnostics and Tools” section in Bing Webmaster Tools, Verify Bingbot tool lets the webmasters check if the Bing user agent string appearing in the server logs are actually from Bing or not. This can help webmasters determine if someone is hiding their true identity and attacking the site by using Bing’s name. Further, it also helps webmasters who have manually configured IP to whitelist Bingbot on their server. Since Bing does not release the list of IPs, using this tool the webmasters can check whether the IPs allowed in the server belong to Bing and whether they are whitelisting the right set of IPs.  Thus, it is evident that a lot can be done by webmasters to improve Bingbot’s performance on their site using the features in Bing Webmaster Tools. These features were developed and have evolved over the years based on feedback we receive from the webmaster community. So, login to Bing Webmaster Tools now to use the features and let us know what you think.  Thanks! Nikunj Daga Program Manager, Bing Webmaster Tools

bingbot Series: JavaScript, Dynamic Rendering, and Cloaking. Oh My!

Bing's Webmaster Blog -

Last week, we posted the second blog of our bingbot Series: Optimizing Crawl Frequency. Today is Halloween and like every day, our crawler (also known as a "spider") is wandering outside, browsing the world wide web, following links, seeking to efficiently discover, index and refresh the best web content for our Bing users.   Occasionally, bingbot encounters websites relying on JavaScript to render their content. Some of these sites link to many JavaScript files that need to be downloaded from the web server. In this setup, instead of making only one HTTP request per page, bingbot has to do several requests. Some some sites are spider traps, with dozens of HTTP calls required to render each page! Yikes. That's not optimal, now is it?  As we shared last week at SMX East, bingbot is generally able to render JavaScript. However, bingbot does not necessarily support all the same JavaScript frameworks that are supported in the latest version of your favorite modern browser. Like other search engine crawlers, it is difficult for bingbot to process JavaScript at scale on every page of every website, while minimizing the number of HTTP requests at the same time.  Therefore, in order to increase the predictability of crawling and indexing by Bing, we recommend dynamic rendering as a great alternative for websites relying heavily on JavaScript. Dynamic rendering is about detecting user agent and rendering content differently for humans and search engine crawlers. We encourage detecting our bingbot user agent, prerendering the content on the server side and outputting static HTML for such sites, helping us minimize the number of HTTP requests and ensure we get the best and most complete version of your web pages every time bingbot visits your site. Is using JavaScript for Dynamic Rendering considered Cloaking? When it comes to rendering content specifically for search engine crawlers, we inevitably get asked whether this is considered cloaking... and there is nothing scarier for the SEO community than getting penalized for cloaking, even during Halloween! The good news is that as long as you make a good faith effort to return the same content to all visitors, with the only difference being the content is rendered on the server for bots and on the client for real users, this is acceptable and not considered cloaking. So if your site relies a lot of JavaScript and you want to improve your crawling and indexing on Bing, look into dynamic rendering: you will certainly benefit immensely, receiving only treats and no tricks! Happy Halloween! Fabrice Canel and Frédéric Dubut Program Managers Microsoft - Bing

bingbot Series: Optimizing Crawl Frequency

Bing's Webmaster Blog -

Last week, we posted the first blog of our bingbot Series: Maximizing Crawl Efficiency highlighting the main goal for bingbot and its core metrics: Crawling Efficiency.   In part 2 of our bingbot series Principal Software Engineering Managing of the crawl team Cheng Lu shares one example of how we’ve optimized our processes to maximize crawl efficiency for large web sites whose content remains static, or unchanging.   Keeping Indexed Content Current and limiting crawling on content that has changed   When most people conduct a search they typically are looking for the most recent content published; however, the search engine results may link to webpages that were published days ago to years ago. This is a challenge, especially when searchers are wanting to keep up with breaking news and the latest trends by accessing the most up-to-date content online. The internet and search index are full of ghosts of yester-years past, that are often resurrected by the power of search engines. For instance, I was able to retrieve the Microsoft 1996 annual report. Interesting, yes, especially if I need to do a historical report, but if I'm looking for the current annual investment report, it is not so useful. The crawler also needs to have discovered, crawled and indexed the latest Microsoft annual report in order for me to discover it when I do a search. The challenge for bingbot is that it can't fetch a web page only once. Once a page is published, the search engine must fetch the page regularly to verify that the content has not been updated and that the page is not a dead link. Defining when to fetch the web page next is the hard problem we are looking to optimize with your help.   Case Study: Cornell University Library - A great source of knowledge with many static, unchanging web pages   One challenge that we are trying to address is how often should bingbot crawl a site to fetch the content. The answer depends on the frequency of which the content is edited and updated.   The Cornell University Library empowers Cornell's research and learning community with deep expertise, innovative services, and outstanding collections strengthened by strategic partnerships. Their web site https://arxiv.org/ is a mine of relevant information and it contains millions of web pages on a range of topic from Physics, to Science to Economics. Not only do they have millions of webpages and PDF files related to computer science, they even have content related to crawling and indexing websites.    Identifying patterns to allow bingbot to reduce crawl frequency   While new web pages may be posted daily, and some pages are updated on a regular basis, most of the content within the Cornell University Library is not edited for month and even years. The content is by in large static and unedited. By unedited, I mean that the HTML may change a little, for example {copyright 2018} will {become 2019} on January 1st,  the CSS and style sheet may change a little; however such changes are not relevant for updating the indexed content within Bing. The content of the page is still the same. Additionally, only few articles are deleted every year. Their library index increases in size with new and updated research, without substantially changing the content of the historically indexed research. Reviewing our crawling data, we discovered that bingbot was over-crawling the content and we were using more resources then necessary to check and re-check that the historical pages maintained static in nature. What we learned was that we could optimize our system to avoid fetching the same content over and over, and instead check periodically for major changes.  This resulted in about 40% crawl saving on this site! While our work in identified patterns for largely static content identified an opportunity to reduce crawling for this “class” of websites (slow and rarely changing content) and in the following posts we’ll share more learnings. Our work with improving crawler efficiency is not done yet, and we’ve got a lot of opportunity ahead of us to continue to improve our crawler’s efficiency and abilities across the hundreds of different types of data that are used to evaluate our crawler scheduling algorithms. The next step is to continue to identify patterns that apply to a multitude of websites, so we can scale our efforts and be more efficient with crawling everywhere.   Stay tuned! Next in this series of posts related to bingbot and our crawler, we’ll provide visibility on the main criteria involved in defining bingbots Crawl Quota and Crawl Frequency per site. I hope you are still looking forward to learning more about how we improve crawl efficiency and as always, we look forward to seeing your comments and feedback.   Thanks! Cheng Lu Principal Software Engineering Manager Microsoft - Bing Fabrice Canel Principal Program Manager Microsoft - Bing  

bingbot Series: Maximizing Crawl Efficiency

Bing's Webmaster Blog -

At the SMX Advanced conference in June, I announced that over the next 18 months my team will focus on improving our Bing crawler bingbot . I asked the audience to share data helping us to optimize our plans. First, I want to say "Thank you"  to those of you who responded and provided us with great insights. Please keep them coming!  To keep you informed of the work we've done so far, we are starting this series of blog posts related to our crawler, bingbot. In this series we will share best practices, demonstrate improvements, and unveil new crawler abilities. Before drilling into details about how our team is continuing to improve our crawler, let me explain why we need bingbot and how we measure bingbot's success. First things first: What is the goal of bingbot? Bingbot is Bing's crawler, sometimes also referred to as a "spider". Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing's searchable index. Its primary goal is to maintain a comprehensive index updated with fresh content. Bingbot uses an algorithm to determine which sites to crawl, how often, and how many pages to fetch from each site. The goal is to minimize bingbot crawl footprint on your web sites while ensuring that the freshest content is available. How do we do that? The algorithmic process selects URLs to be crawled by prioritizing relevant known URLs that may not be indexed yet, and URLs that have already been indexed that we are checking for updates to ensure that the content is still valid (example not a dead link) and that it has not changed. We also crawl content specifically to discovery links to new URLs that have yet to be discovered. Sitemaps and RSS/Atom feeds are examples of URLs fetched primarily to discovery new links. Measuring bingbot success : Maximizing Crawl efficiency Bingbot crawls billions of URLs every day. It's a hard task to do this at scale, globally, while satisfying all webmasters, web sites, content management systems, whiling handling site downtimes and ensuring that we aren't crawling too frequently or often. We've heard concerns that bingbot doesn't crawl frequently enough and their content isn't fresh within the index; while at the same time we've heard that bingbot crawls too often causing constraints on the websites resources. It's an engineering problem that hasn't fully been solved yet. Often, the issue is in managing the frequency that bingbot needs to crawl a site to ensure new and updated content is included in the search index. Some webmasters request to have their sites crawled daily by the bingbot to ensure that Bing has the freshest version of their site in the index;  whereas the majority of webmasters would prefer to only have bingbot crawl their site when new URLs have been added or content has been updated and changed. The challenge we face, is how to model the bingbot algorithms based on both what a webmaster wants for their specific site, the frequency in which content is added or updated, and how to do this at scale. To measure how smart our crawler is, we measure bingbot crawl efficiency. The crawl efficiency is how often we crawl and discover new and fresh content per page crawled.  Our crawl efficiency north star is to crawl an URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links) . The more we crawl duplicated, unchanged content, the lower our Crawl Efficiency metric is. Later this month, Cheng Lu, our engineer lead for the crawler team, will continue this series of blog posts by sharing examples of how the Crawl Efficiency has improved over the last few months. I hope you are looking forward to learning more about how we improve crawl efficiency and as always, we look forward to seeing your comments and feedback. Thanks! Fabrice Canel Principal Program Manager, Webmaster Tools Microsoft - Bing  

Introducing Bing AMP viewer and Bing AMP cache

Bing's Webmaster Blog -

In 2016, Bing joined the Accelerated Mobile Pages (AMP for short) open-source effort to help you “find” and “do” searches faster, regardless of where you are and on any device when you are looking for informaiton. Today, we are pleased to announce the release of Bing AMP viewer and Bing AMP Cache enabling AMP-enabled web pages to work directly from Bing’s mobile search results allowing Bing to provide faster mobile experiences to Bing users. On Monday, the 17th we started the first phase of the global roll out of this AMP viewer and AMP carrousel in the United States for the news carrousel. We will continue the phased roll out to more web sites, more countries and regions and other links in the search results pages. Also, if you are in the United States, try it out on your mobile device by navigating to https://www.bing.com and search for news related queries and tapping the search results labelled with the AMP icon: . Advice for AMP webmasters, AMP advertisers The AMP protocol offers the ability to cache and serve cached copied AMP content that is published on the web, providing faster user experiences on Bing. In order to enable your AMP published content within Bing, you need to allow the Bingbot (our crawler) to fetch AMP content and allow cross-origin resource sharing (CORS) for bing-amp.com domain. Most AMP enabled sites and advertisers have already authorized the CORS sharing for the ampproject.org domain, but now need to also bing-amp.com to the allowed list.  Thank you,  Fabrice Canel Principal Program Manager Microsoft Bing  

Anonymous URL Submission Tool Being Retired

Bing's Webmaster Blog -

Saying Goodbye is never easy, but the time has come to announce the withdrawal of anonymous non-signed in support Bing's URL submission tool. Webmaster will still be able to log in and access Submit URL tool in Bing Webmaster Tools, and this is easier than ever as the tool now supports Google and Facebook authentication in addition to existing Microsoft accounts. Why say goodbye? Well, the URLs received are by far too low quality to be trustable, and webmasters preferring having more ownership of the URLs for their site. In order to use the tool, webmasters just need to login, add and verify their site. Then navigate to the Submit URL tool within the Configure My Site menu options. In case the webmasters want to use our Bing Webmaster tools API, webmasters have to generate an API key through Bing Webmaster Tools and follow the guidelines for its usage here. In case you haven't signed up on the tool yet, please click here to sign up. Thank you, The Bing Webmaster Tools Team

Collaboration and user management in the new Search Console

Google Webmaster Central Blog -

As part of our reinvention of Search Console, we have been rethinking the models of facilitating cooperation and accountability for our users. We decided to redesign the product around cooperative team usage and transparency of action history. The new Search Console will gradually provide better history tracking to show who performed which significant property-affecting modifications, such as changing a setting, validating an issue or submitting a new sitemap. In that spirit we also plan to enable all users to see critical site messages. New featuresUser management is now an integral part of Search Console.The new Search Console enables you to share a read-only view of many reports, including Index coverage, AMP, and Mobile Usability. Learn more.A new user management interface that enables all users to see and (if appropriate), manage user roles for all property users. New Role definitionIn order to provide a simpler permission model, we are planning to limit the "restricted" user role to read-only status. While being able to see all information, read-only users will no longer be able to perform any state-changing actions, including starting a fix validation or sharing an issue. Best practicesAs a reminder, here are some best practices for managing user permissions in Search Console: Grant users only the permission level that they need to do their work. See the permissions descriptions.If you need to share an issue details report, click the Share link on that page.Revoke permissions from users who no longer work on a property.When removing a previous verified owner, be sure to remove all verification tokens for that user.Regularly audit and update the user permissions using the Users & Permissions page in new Search Console. User feedbackAs part of our Beta exploration, we released visibility of the user management interface to all user roles. Some users reached out to request more time to prepare for the updated user management model, including the ability of restricted and full users to easily see a list of other collaborators on the site. We’ve taken that feedback and will hold off on that part of the launch. Stay tuned for more updates relating to collaboration tools and changes on our permission models. As always, we love to hear feedback from our users. Feel free to use the feedback form within Search Console, and we welcome your discussions in our help forums as well! Posted by John Mueller, Google Switzerland

Links, Mobile Usability, and site management in the new Search Console

Google Webmaster Central Blog -

More features are coming to the new Search Console. This time we've focused on importing existing popular features from the old Search Console to the new product. Links Report Search Console users value the ability to see links to and within their site, as Google Search sees them. Today, we are rolling out the new Links report, which combines the functionality of the “Links to your site” and “Internal Links” reports on the old Search Console. We hope you find this useful! Mobile Usability report Mobile Usability is an important priority for all site owners. In order to help site owners with fixing mobile usability issues, we launched the Mobile Usability report on the new Search Console. Issue names are the same as in the old report but we now allow users to submit a validation and reindexing request when an issue is fixed, similar to other reports in the new Search Console. Site and user management To make the new Search Console feel more like home, we’ve added the ability to add and verify new sites, and manage your property's users and permissions, directly in new Search Console using our newly added settings page. Keep sending feedback As always, we would love to get your feedback through the tools directly and our help forums so please share and let us know how we're doing.Posted by Ariel Kroszynski and Roman Kecher - Search Console engineers

Introducing JSON-LD Support in Bing Webmaster Tools

Bing's Webmaster Blog -

Bing is proud to introduce JSON-LD support as part of Bing Webmaster Tools, as announced at the 2018 SMX Advanced. Users can login to Bing Webmaster Tools and validate their JSON-LD implementation through the Markup Validator Tool present in the Diagnostics and Tools section. After the inclusion of JSON-LD support, the Markup Validator now supports seven markup languages, including Schema.org, HTML Microdata, Microformats, Open Graph and RDFa.  Bing works hard to understand the content of a page and one of the clues that Bing uses is structured data. JSON-LD, or JavaScript Object Notation for Linked Data, is an extension of JSON-based data format that can be used to implement structured data on your site so Bing and other search engines can better understand the content on your site. One of the advantages is that JSON-LD can be implemented without modifying the HTML content of your pages and can be hidden in the header, body or foot of the page. This effectively means that webmasters can go about designing their pages as they like without having to worry about arranging the information for markup implementations. JSON-LD makes defining links and relationships between data and entities between the data present on your pages easy because it supports nested data. However, with the wide array of possibilities that comes with JSON-LD, the webmasters should be very alert as to not put invalid and incorrect information in the markup. Remember, even though the markup is not visible on your page, it is still read by the search engines and putting spam data in the markup can hamper your presence on the search engines. Let us know what you think of the JSON-LD support. If you don’t have a Webmaster Tools account, you can sign up today.   Thank you, The Bing Webmaster Team  

Intermittent Webmaster Tools API Issues Resolved

Bing's Webmaster Blog -

The Bing Webmaster team recently received feedback that our APIs were intermittently failing, and we deeply regret any inconveniences caused from the API failures. We recognize the frustrations that this may have caused.  Upon investigation, we discovered a technical glitch which led to API call failure that is now resolved. We are very grateful to you, our users, who brought this to our attention and thank you for your continued feedback and support. We're Listening Bing and Bing Webmaster Tools are actively listening to you and we value your feedback. It’s important to how we continually improve Bing and to help notify us of potential issues. It’s easy to provide feedback: just look for the Feedback button or link at the bottom of each page. It’s in the footer or the lower-right corner and it looks something like this: We are using advances in technology to make it easier to quickly find what you are looking for – from answers to life's big questions or an item in an image you want to learn more about. At Bing, our goal is to help you get answers with less effort.  We appreciate your feedback and the more that you can send, the more we can use it to improve Bing. Have a suggestion? Tell us! The more feedback the merrier. Please let us know. The Bing Webmaster Tools Team

Hey Google, what's the latest news?

Google Webmaster Central Blog -

Since launching the Google Assistant in 2016, we have seen users ask questions about everything from weather to recipes and news. In order to fulfill news queries with results people can count on, we collaborated on a new schema.org structured data specification called speakable for eligible publishers to mark up sections of a news article that are most relevant to be read aloud by the Google Assistant. When people ask the Google Assistant -- "Hey Google, what's the latest news on NASA?", the Google Assistant responds with an excerpt from a news article and the name of the news organization. Then the Google Assistant asks if the user would like to hear another news article and also sends the relevant links to the user's mobile device. As a news publisher, you can surface your content on the Google Assistant by implementing Speakable markup according to the developer documentation. This feature is now available for English language users in the US and we hope to launch in other languages and countries as soon as a sufficient number of publishers have implemented speakable. As this is a new feature, we are experimenting over time to refine the publisher and user experience. If you have any questions, ask us in the Webmaster Help Forum. We look forward to hearing from you! Posted by TV Raman, Senior Staff Software Engineer

An update to referral source URLs for Google Images

Google Webmaster Central Blog -

Every day, hundreds of millions of people use Google Images to visually discover and explore content on the web. Whether it be finding ideas for your next baking project, or visual instructions on how to fix a flat tire, exploring image results can sometimes be much more helpful than exploring text. Updating the referral sourceFor webmasters, it hasn't always been easy to understand the role Google Images plays in driving site traffic. To address this, we will roll out a new referrer URL specific to Google Images over the next few months. The referrer URL is part of the HTTP header, and indicates the last page the user was on and clicked to visit the destination webpage. If you create software to track or analyze website traffic, we want you to be prepared for this change. Make sure that you are ingesting the new referer URL, and attribute the traffic to Google Images. The new referer URL is: https://images.google.com. If you use Google Analytics to track site data, the new referral URL will be automatically ingested and traffic will be attributed to Google Images appropriately. Just to be clear, this change will not affect Search Console. Webmasters will continue to receive an aggregate list of top search queries that drive traffic to their site. How this affects country-specific queriesThe new referer URL has the same country code top level domain (ccTLD) as the URL used for searching on Google Images. In practice, this means that most visitors worldwide come from images.google.com. That's because last year, we made a change so that google.com became the default choice for searchers worldwide. However, some users may still choose to go directly to a country specific service, such as google.co.uk for the UK. For this use case, the referer uses that country TLD (for example, images.google.co.uk). We hope this change will foster a healthy visual content ecosystem. If you're interested in learning how to optimize your pages for Google Images, please refer to the Google Image Publishing Guidelines. If you have questions, feedback or suggestions, please let us know through the Webmaster Tools Help Forum. Posted by Ashutosh Agarwal, Product Manager, Google Images

How we fought webspam - Webspam Report 2017

Google Webmaster Central Blog -

We always want to make sure that when you use Google Search to find information, you get the highest quality results. But, we are aware of many bad actors who are trying to manipulate search ranking and profit from it, which is at odds with our core mission: to organize the world's information and make it universally accessible and useful. Over the years, we've devoted a huge effort toward combating abuse and spam on Search. Here's a look at how we fought abuse in 2017. We call these various types of abuse that violate the webmaster guidelines “spam.” Our evaluation indicated that for many years, less than 1 percent of search results users visited are spammy. In the last couple of years, we’ve managed to further reduce this by half. Google webspam trends and how we fought webspam in 2017As we continued to improve, spammers also evolved. One of the trends in 2017 was an increase in website hacking—both for spamming search ranking and for spreading malware. Hacked websites are serious threats to users because hackers can take complete control of a site, deface homepages, erase relevant content, or insert malware and harmful code. They may also record keystrokes, stealing login credentials for online banking or financial transactions. In 2017 we focused on reducing this threat, and were able to detect and remove from search results more than 80 percent of these sites. But hacking is not just a spam problem for search users—it affects the owners of websites as well. To help website owners keep their websites safe, we created a hands-on resource to help webmasters strengthen their websites’ security and revamped our help resources to help webmasters recover from a hacked website. The guides are available in 19 languages.We’re also recognizing the importance of robust content management systems (CMSs). A large percentage of websites are run on one of several popular CMSs, and subsequently spammers exploited them by finding ways to abuse their provisions for user-generated content, such as posting spam content in comment sections or forums. We’re working closely with many of the providers of popular content management systems like WordPress and Joomla to help them also fight spammers that abuse their forums, comment sections and websites.Another abuse vector is the manipulation of links, which is one of the foundation ranking signals for Search. In 2017 we doubled down our effort in removing unnatural links via ranking improvements and scalable manual actions. We have observed a year-over-year reduction of spam links by almost half. Working with users and webmasters for a better webWe’re here to listen: Our automated systems are constantly working to detect and block spam. Still, we always welcome hearing from you when something seems … phishy. Last year, we were able to take action on nearly 90,000 user reports of search spam.Reporting spam, malware and other issues you find helps us protect the site owner and other searchers from this abuse. You can file a spam report, a phishing report or a malware report. We very much appreciate these reports—a big THANK YOU to all of you who submitted them.We also actively work with webmasters to maintain the health of the web ecosystem. Last year, we sent 45 million messages to registered website owners via Search Console letting them know about issues we identified with their websites. More than 6 million of these messages are related to manual actions, providing transparency to webmasters so they understand why their sites got manual actions and how to resolve the issue.Last year, we released a beta version of a new Search Console to a limited number of users and afterwards, to all users of Search Console. We listened to what matters most to the users, and started with popular functionalities such as Search performance, Index Coverage and others. These can help webmasters optimize their websites' Google Search presence more easily.Through enhanced Safe Browsing protections, we continue to protect more users from bad actors online. In the last year, we have made significant improvements to our safe browsing protection, such as broadening our protection of macOS devices, enabling predictive phishing protection in Chrome, cracked down on mobile unwanted software, and launched significant improvements to our ability to protect users from deceptive Chrome extension installation.We have a multitude of channels to engage directly with webmasters. We have dedicated team members who meet with webmasters regularly both online and in-person. We conducted more than 250 online office hours, online events and offline events around the world in more than 60 cities to audiences totaling over 220,000 website owners, webmasters and digital marketers. In addition, our official support forum has answered a high volume of questions in many languages. Last year, the forum had 63,000 threads generating over 280,000 contributing posts by 100+ Top Contributors globally. For more details, see this post. Apart from the forums, blogs and the SEO starter guide, the Google Webmaster YouTube channel is another channel to find more tips and insights. We launched a new SEO snippets video series to help with short and to-the-point answers to specific questions. Be sure to subscribe to the channel!Despite all these improvements, we know we’re not yet done. We’re relentless in our pursue of an abuse-free user experience, and will keep improving our collaboration with the ecosystem to make it happen.Posted by Cody Kwok, Principal Engineer

Introducing the Indexing API for job posting URLs

Google Webmaster Central Blog -

Last June we launched a job search experience that has since connected tens of millions of job seekers around the world with relevant job opportunities from third party providers across the web. Timely indexing of new job content is critical because many jobs are filled relatively quickly. Removal of expired postings is important because nothing's worse than finding a great job only to discover it's no longer accepting applications. Today we're releasing the Indexing API to address this problem. This API allows any site owner to directly notify Google when job posting pages are added or removed. This allows Google to schedule job postings for a fresh crawl, which can lead to higher quality user traffic and job applicant satisfaction. Currently, the Indexing API can only be used for job posting pages that include job posting structured data. For websites with many short-lived pages like job postings, the Indexing API keeps job postings fresh in Search results because it allows updates to be pushed individually. This API can be integrated into your job posting flow, allowing high quality job postings to be searchable quickly after publication. In addition, you can check the last time Google received each kind of notification for a given URL. Follow the Quickstart guide to see how the Indexing API works. If you have any questions, ask us in the Webmaster Help Forum. We look forward to hearing from you! Posted by Zach Clifford, Software Engineer

New URL inspection tool & more in Search Console

Google Webmaster Central Blog -

A few months ago, we introduced the new Search Console. Here are some updates on how it's progressing. Welcome "URL inspection" tool One of our most common user requests in Search Console is for more details on how Google Search sees a specific URL. We listened, and today we've started launching a new tool, “URL inspection,” to provide these details so Search becomes more transparent. The URL Inspection tool provides detailed crawl, index, and serving information about your pages, directly from the Google index. Enter a URL that you own to learn the last crawl date and status, any crawling or indexing errors, and the canonical URL for that page. If the page was successfully indexed, you can see information and status about any enhancements we found on the page, such as linked AMP version or rich results like Recipes and Jobs. URL is indexed with valid AMP enhancement If a page isn't indexed, you can learn why. The new report includes information about noindex robots meta tags and Google's canonical URL for the page. URL is not indexed due to ‘noindex’ meta tag in the HTML A single click can take you to the issue report showing all other pages affected by the same issue to help you track down and fix common bugs. We hope that the URL Inspection tool will help you debug issues with new or existing pages in the Google Index. We began rolling it out today; it will become available to all users in the coming weeks. More exciting updates In addition to the launch of URL inspection, we have a few more features and reports we recently launched to the new Search Console: Sixteen months of traffic data: The Search Analytics API now returns 16 months of data, just like the Performance report. Recipe report: The Recipe report help you fix structured data issues affecting recipes rich results. Use our task-oriented interface to test and validate your fixes; we will keep you informed on your progress using messages. New Search Appearance filters in Search Analytics: The performance report now gives you more visibility on new search appearance results, including Web Light and Google Play Instant results. Thank you for your feedback We are constantly reading your feedback, conducting surveys, and monitoring usage statistics of the new Search Console. We are happy to see so many of you using the new issue validation flow in Index Coverage and the AMP report. We notice that issues tend to get fixed quicker when you use these tools. We also see that you appreciate the updates on the validation process that we provide by email or on the validation details page. We want to thank everyone who provided feedback: it has helped us improve our flows and fix bugs on our side. More to come The new Search Console is still beta, but it's adding features and reports every month. Please keep sharing your feedback through the various channels and let us know how we're doing.Posted by Roman Kecher and Sion Schori - Search Console engineers

Google Search at I/O 2018

Google Webmaster Central Blog -

With the eleventh annual Google I/O wrapped up, it’s a great time to reflect on some of the highlights.What we did at I/OThe event was a wonderful way to meet many great people from various communities across the globe, exchange ideas, and gather feedback. Besides many great web sessions, codelabs, and office hours we shared a few things with the community in two sessions specific to Search:Deliver search-friendly JavaScript-powered websites with John Mueller and Tom GreenawayBuild a successful web presence with Google Search with Mariya Moeva and John MuellerThe sessions included the launch of JavaScript error reporting in the Mobile Friendly Test tool, dynamic rendering (we will discuss this in more detail in a future post), and an explanation of how CMS can use the Indexing and Search Console APIs to provide users with insights. For example, Wix lets their users submit their homepage to the index and see it in Search results instantly, and Squarespace created a Google Search keywords report to help webmasters understand what prospective users search for.During the event, we also presented the new Search Console in the Sandbox area for people to try and were happy to get a lot of positive feedback, from people being excited about the AMP Status report to others exploring how to improve their content for Search.Hands-on codelabs, case studies and moreWe presented the Structured Data Codelab that walks you through adding and testing structured data. We were really happy to see that it ended up being one of the top 20 codelabs by completions at I/O. If you want to learn more about the benefits of using Structured Data, check out our case studies.During the in-person office hours we saw a lot of interest around HTTPS, mobile-first indexing, AMP, and many other topics. The in-person Office Hours were a wonderful addition to our monthly Webmaster Office Hours hangout. The questions and comments will help us adjust our documentation and tools by making them clearer and easier to use for everyone.Highlights and key takeawaysWe also repeated a few key points that web developers should have an eye on when building websites, such as:Indexing and rendering don’t happen at the same time. We may defer the rendering to a later point in time.Make sure the content you want in Search has metadata, correct HTTP statuses, and the intended canonical tag.Hash-based routing (URLs with "#") should be deprecated in favour of the JavaScript History API in Single Page Apps.Links should have an href attribute pointing to a URL, so Googlebot can follow the links properly.Make sure to watch this talk for more on indexing, dynamic rendering and troubleshooting your site. If you wanna learn more about things to do as a CMS developer or theme author or Structured Data, watch this talk.We were excited to meet some of you at I/O as well as the global I/O extended events and share the latest developments in Search. To stay in touch, join the Webmaster Forum or follow us on Twitter, Google+, and YouTube. Posted by Martin Splitt, Webmaster Trends Analyst

Our goal: helping webmasters and content creators

Google Webmaster Central Blog -

Great websites are the result of the hard work of website owners who make their content and services accessible to the world. Even though it’s simpler now to run a website than it was years ago, it can still feel like a complex undertaking. This is why we invest a lot of time and effort in improving Google Search so that website owners can spend more time focusing on building the most useful content for their users, while we take care of helping users find that content. Most website owners find they don’t have to worry much about what Google is doing—they post their content, and then Googlebot discovers, crawls, indexes and understands that content, to point users to relevant pages on those sites. However, sometimes the technical details still matter, and sometimes a great deal.For those times when site owners would like a bit of help from someone at Google, or an explanation for why something works a particular way, or why things appear in a particular way, or how to fix what looks like a technical glitch, we have a global team dedicated to making sure there are many places for a website owner to get help from Google and knowledgeable members of the community.The first place to start for help is Google Webmasters, a place where all of our support resources (many of which are available in 40 languages) are within easy reach:Google Web Fundamentals: Provides technical guidance on building a modern website that takes advantage of open web standards.Google Search developer documentation: Describes how Google crawls and indexes a website. Includes authoritative guidance on building a site that is optimized for Google Search. Search Console Help Center: Provides detailed information on how to use and take advantage of Search Console, the best way for a website owner to understand how Google sees their site. The Search Engine Optimization (SEO) Starter Guide: Provides a complete overview of the basics of SEO according to our recommended best practices.Google webmaster guidelines: Describes policies and practices that may lead to a site being removed entirely from the Google index or otherwise affected by an algorithmic or manual spam action that can negatively affect their Search appearance. Google Webmasters YouTube ChannelOur second path to getting help is through our Google Webmaster Central Help Forums. We have forums in 16 languages—in English, Spanish, Hindi, French, Italian, Portuguese, Japanese, German, Russian, Turkish, Polish, Bahasa Indonesia, Thai, Vietnamese, Chinese and Korean. The forums are staffed with dedicated Googlers who are there to make sure your questions get answered. Aside from the Googlers who monitor the forums, there is an amazing group of Product Experts who generously offer their time to help other members of the community—many times providing greater detail and analysis for a particular website’s content than we could. The forums allow for both a public discussion and, if the case requires it, for private follow-up replies in the forum.A third path for support to website owners is our series of Online Webmaster Office Hours — in English, German, Japanese, Turkish, Hindi and French. Anyone who joins these is welcome to ask us questions about website appearance in Google Search, which we will answer to the best of our abilities. All of our team members think that one of the best parts of speaking at conferences and events is the opportunity to answer questions from the audience,  and the online office hours format creates that opportunity for many more people who might not be able to travel to a specialized event. You can always check out the Google Webmaster calendar for upcoming webmaster officer hours and live events.While how a website behaves on the web is openly visible to all who can see it, we know that some website owners prefer not to make it known their website has a problem in a public forum. There’s no shame in asking for support, but if you have an issue for your website that seems sensitive—for which you don’t think you can share all the details publicly—you can call out that you would prefer to share necessary details only with someone experienced and who is willing to help, using the forum’s “Private Reply” feature.Are there other things you think we should be doing that would help your website get the most out of search? Please let us know -- in our forums, our office hours, or via Twitter @googlewmc.Posted by Juan Felipe Rincón from Google’s Webmaster Outreach & Support team

Google I/O 2018 - What sessions should SEOs and Webmasters watch live ?

Google Webmaster Central Blog -

Google I/O 2018 is starting today in California, to an international audience of 7,000+ developers. It will run until Thursday night. It is our annual developers festival, where product announcements are made, new APIs and frameworks are introduced, and Product Managers present the latest from Google.However, you don't have to physically attend the event to take advantage of this once-a-year opportunity: many conferences and talks are live streamed on YouTube for anyone to watch. You will find the full-event schedule here.Dozens upon dozens of talks will take place over the next 3 days. We have hand picked the talks that we think will be the most interesting for webmasters and SEO professionals. Each link shared will bring you to pages with more details about each talk, and you will find out how to tune in to the live stream. All times are California time (PCT). We might add other sessions to this list.Tuesday, May 8th3pm - Web Security post Spectre/Meltdown, with Emily Schechter and Chris Palmer - more info.5pm - Dru Knox and Stephan Somogyi talk about building a seamless web with Chrome - more info.Wednesday, May 9th9.30am - Ewa Gasperowicz and Addy Osmani talk about Web Performance and increasing control over the loading experience - more info.10.30am - Alberto Medina and Thierry Muller will explain how to make a WordPress site progressive - more info.11.30am - Rob Dodson and Dominic Mazzoni will cover "What's new in web accessibility" - more info.3.30pm - Michael Bleigh will introduce how to leverage AMP in Firebase for a blazing fast website - more info.4.30pm - Rick Viscomi and Vinamrata Singal will introduce the latest with Lighthouse and Chrome UX Report for Web Performance - more info.Thursday, May 10th8.30am - John Mueller and Tom Greenaway will talk about building Search-friendly JavaScript websites - more info.9.30am - Build e-commerce sites for the modern web with AMP, PWA, and more, with Adam Greenberg and Rowan Merewood - more info.12.30pm - Session on "Building a successful web presence with Google Search" by John Mueller and Mariya Moeva - more info.This list is only a sample of the content at this year's Google I/O, and there might be many more that are interesting to you! To find out about those other talks, check out the full list of web sessions, but also the sessions about Design, the Cloud sessions, the machine learning sessions, and more… We hope you can make the time to watch the talks online, and participate in the excitement of I/O ! The videos will also be available on Youtube after the event, in case you can't tune in live.Posted by Vincent Courson, Search Outreach Specialist, and the Google Webmasters team

Pages

Recommended Content

Subscribe to Complete Hosting Guide aggregator - Search Engine Blogs