What is the GreenWebChecker and why is it hitting my site? The GreenWebChecker is part of the Green Web Foundation’s Green Web Check service and Browser extension, which users can use to find out if a given site runs on 100% green energy. When a user looks up your site, we perform two GET requests to look for a carbon.txt file, the response of this request is then cached for 24 hours. You should never see GreenWebChecker request any other paths on your server, or make more than 2 requests a day. For further information, read on!
What is GreenWebChecker/1.0.0?
If you’ve arrived at this page, it’s likely because you’ve seen requests from the User Agent “GreenWebChecker/1.0.0” in your web server logs, and are wondering what they are, and where they’re from.
GreenWebChecker is a service operated by the Green Web Foundation in order to power our Green Web Checker: when a user carries out a green check against a domain, our service tries to match that domain to a known green hosting provider by one of three methods: by matching a known green IP range, a known green ASN, or by finding a carbon.txt file being served over http at the domain: The GreenWebChecker user agent is used for this last request – to fetch the carbon.txt file, and only for this request – It will only ever hit the /carbon.txt and /.well-known/carbon.txt paths, with a HEAD or GET request, and won’t touch any other part of your site.
How often does the Green Web Checker hit my site?
When we look for a carbon.txt at a domain, we cache the result for 24 hours, whether we successfully find a carbon.txt file or not. Therefore, in any single 24-hour period, you should see a maximum of two request from GreenWebChecker/1.0: one for /carbon.txt, and another for /.well-known/carbon.txt – the two locations where we check for a carbon.txt file.
How can I make my provider show as green in the Green Web Checker?
If your provider is run on 100% renewable energy, and you’re not yet listed with us, it’s very easy to get verified! Follow the link for more information and to get started, or get in touch with us. Verifying with us means that the sites you host will show as green in the green web checker, and will also allow you to be listed in our green hosting directory.
What is gwf-carbon-txt-crawler?
The carbon.txt crawler is used to monitor adoption of carbon.txt, a Green Web Foundation project to create a standardised approach for making sustainability data transparent. Every year we crawl the web, looking for carbon.txt files, in order to measure adoption of the standard. We don’t crawl any other pages, and don’t store any data apart from the carbon.txt files we find.
How often does the Carbon.txt Crawler hit my site?
We run the carbon.txt crawl annually, checking the top 100 million hostname in the Common Crawl Web Graph. For each of these, we make a small number of HTTP requests – one for the /robots.txt (which we respect – we won’t proceed to crawl your site if robots.txt disallows it), one for the root page of the domain at /, one each for the two carbon.txt locations – /carbon.txt and /.well-known/carbon.txt, and another for any URLs found in the carbon.txt DNS record or HTTP header, if present.
If a carbon.txt file is found, we store it, but do not store any other data gathered from your site. If you are a hosting provider who hosts multiple domains, or you have multiple subdomains, you will see requests for each of the domains you host, but we attempt to space requests for the same TLD out in order to ensure we’re not repeatedly hitting the same servers.
If you don’t want to take part in this crawl, you can add the following block to your robots.txt file:
User-agent: gwf-carbon-txt-crawler
Disallow: /If you are interested in adopting carbon.txt on your site, you can find all the details, implementation instructions, and tools to build and verify your carbon.txt file at https://carbontxt.org.
I have an issue or question which isn’t listed here.
If you have any other comments, questions or issues, please don’t hesitate to reach out to us via our support form, and we will get back to you as soon as possible.