robots.txt
# pages-helpm
mapple
05/10/2023, 6:49 AMHi - I'm moving a previous wordpress site to a static site using pages and a cloudflare custom domain. I've added a robots.txt file to the files uploaded - when I visit mysite.com/robots.txt or mysite.com/anystring it's showing the index page - how do I stop that behaviour - I'd rather the user/crawler was served a 404 for mysite.com/anystring than the home page?
w
Walshy | Pages
05/10/2023, 8:10 AMm
mapple
05/10/2023, 8:40 AMthanks that's really useful - I've added a 404 and it's all behaving better - I'm still getting the index page served on the custom domain site.com when I try and view site.com/robots.txt .... (I have added robots.txt to the root folder I upload to deploy and can see it there in the uploaded files) how can I get pages to not deisplay the index page if googlebot comes looking for robots.txt?
mapple
05/10/2023, 9:04 AMI've carried on digging around - am I going to need something like proxy flare to serve the robots.txt? https://developers.cloudflare.com/pages/platform/functions/plugins/community-plugins/
w
Walshy | Pages
05/10/2023, 9:08 AMNope, it should just serve it
Walshy | Pages
05/10/2023, 9:09 AMCan you take a screenshot of the "Assets uploaded" tab in the dashboard?
m
mapple
05/10/2023, 9:16 AMI can give account ID in case it's something with the custom domain that's causing it - didn't want to share url publicy
https://cdn.discordapp.com/attachments/1105748534847873114/1105785329258070067/assets-uploads.png▾
w
Walshy | Pages
05/10/2023, 9:17 AMHmm, yeah do you want to DM me your account ID + domain
m
mapple
05/10/2023, 9:17 AMsure
mapple
05/10/2023, 9:23 AMresolved - thanks very much for your help - for anyone in the future reading this double check response with curl https://www.yoursite.com/robots.txt
5 Views