In reply to Lucas Dobre's message of 28 Apr 2025:

Duckduckgo searching for "Boinc config" produces:

"We would like to show you a description here but the site won't allow us."

Why?

You need to ask DuckDuckGo that.

Doing a search on DuckDuckGo using
Boinc config
without the quotation marks gives lots of results.

---

Grant
Darwin NT.

First result is a link to https://boinc.berkeley.edu/wiki/Client_configuration
It has "We would like to show you a description here but the site won't allow us." instead of description.
Same happens with Bing and Yahoo.
Robots.txt looks like this:

User-agent: Google
Disallow:

User-agent: *
Disallow: /trac/
Disallow: /gitweb
Disallow: /git
Disallow: /translate
Disallow: /doxygen
Disallow: /w/
Disallow: /wiki/
Disallow: /dev/
Disallow: /alpha/
Disallow: /cplan/
Disallow: /test/

In reply to Lucas Dobre's message of 28 Apr 2025:

Thanks kotenok. The question is why Boinc puts in a disallow for all those things. If everyone did that search engines would be almost useless.

The search engines are welcome to crawl all the public areas of the site but no site really wants things like the dev and test areas to be indexed and searchable.

Why is /wiki/ disallowed?

Most likely the default option and no one ever got around to changing it.

In reply to kotenok2000's message of 2 May 2025:

Why is /wiki/ disallowed?

/wiki/ anyway is deprecated and was moved to GitHub wiki

---

BOINC maintainer.
For any insight, check my BOINC Development Blog.

I hate the AI part of the new Google search engine, but this time it did help out. (Also added it to Mr Hucker's banishment post)

A robots.txt file is a text file placed in the root directory of a website that provides instructions to web crawlers, such as search engine bots, about which parts of the website they can or cannot access. It's used to control crawling and indexing, not to block pages from appearing in search results.

Control crawling:
It instructs crawlers on which URLs they can or cannot visit, helping to manage bandwidth consumption and prevent overload.

Improve search engine indexing:
By guiding crawlers, it can help ensure that important pages are discovered and indexed efficiently.

Prevent indexing of certain pages:
While not a direct method to prevent a page from being indexed, it can help prevent crawlers from visiting specific pages or directories.

Specify sitemaps:
It can also list one or more XML sitemaps, which are helpful for search engine crawlers.

robots.txt is a suggestion to crawlers, not a guarantee. While major search engines generally respect it, they may choose to ignore it.

robots.txt is not a mechanism for preventing pages from being indexed. To block a page from being indexed, use tags or password-protect the page

In summary, a robots.txt file is a way to communicate with web crawlers about which parts of your website they should or should not access, helping to manage crawling, improve indexing, and potentially prevent certain pages from being crawled, though it's not a definitive method for blocking indexing.