deskeenWeb Builder Module - Create robots.txt file
This module allows the @deskeen/web-builder engine to create a robots.txt file.
Install
npm install @deskeen/web-builder
npm install @deskeen/web-builder-create-robots-txtUsage
And add the module to the list of modules:
const builder = require('@deskeen/web-builder')
await builder.build({
source: [
// List of files or directories
],
modules: [
[
'@deskeen/web-builder-create-robots-txt',
{
path: '/deploy/path', // Deploy path
sitemapUrl: 'https://example.com/sitemap.xml', // Sitemap URL
disallowedCrawlers: [
// List of crawlers to exclude
// Example:
// 'Facebot'
],
disallowedUrls: [
// List of URLs to exclude
// Example:
// '/login'
],
}
]
]
})Popular Crawler Names
- Apple:
Applebot - Amazon:
ia_archiver - Askdotcom:
Teoma - Bing:
bingbot - DuckDuckGo:
DuckDuckBot - Facebook:
Facebot - Google:
Googlebot - Google Ads:
AdsBot-Google,Mediapartners-Google - Google Images:
Googlebot-Image - IBM:
ScoutJet - MSN:
msnbot - Twitter:
Twitterbot - Yahoo:
Slurp - Chinese search engine:
baiduspider - China’s largest eCommerce site:
EtaoSpider - Russian search engine:
Yandex - Czech Republic search engine:
seznambot
Contact
You can reach me at {myfirstname}@{myname}.fr
Licence
MIT Licence - Copyright (c) Morgan Schmiedt