hiding the private thread links from searching engine
Quote from rgn_sss on December 13, 2018, 10:22 pmHello Asgaros,
we faced with a problem of private threads links availability. We have some private chapters and threads which only available for a special group of users. However, forum announces these links to the searching engine systems, so it is indexing successfully, and it’s possible to find such links using searching engines like google, etc. If somebody who has no account joined to the special group tries to open such link, he cannot read the thread for sure. But , but the very fact that private threads links had been shared by forum and indexed by searching engine seems to be bad , cuz it’s possible for “wrong” user to understang the meaning of private topic only by reading it’s header shown in url. Please check if it’s a bug and how to solve it . Thanks
Hello Asgaros,
we faced with a problem of private threads links availability. We have some private chapters and threads which only available for a special group of users. However, forum announces these links to the searching engine systems, so it is indexing successfully, and it’s possible to find such links using searching engines like google, etc. If somebody who has no account joined to the special group tries to open such link, he cannot read the thread for sure. But , but the very fact that private threads links had been shared by forum and indexed by searching engine seems to be bad , cuz it’s possible for “wrong” user to understang the meaning of private topic only by reading it’s header shown in url. Please check if it’s a bug and how to solve it . Thanks
Quote from Asgaros on December 13, 2018, 10:33 pmHello @rgn_sss
Can you give some details where links are leaked to search-engines?
I fixed an issue in the unread-view with the last update which caused that unaccessible topics were listed there. Maybe those links came from this former bug. Currently I am not aware of any other remaining information disclosure-issues.
Hello @rgn_sss
Can you give some details where links are leaked to search-engines?
I fixed an issue in the unread-view with the last update which caused that unaccessible topics were listed there. Maybe those links came from this former bug. Currently I am not aware of any other remaining information disclosure-issues.
Quote from rgn_sss on December 14, 2018, 10:47 amFor example, we have a private topic https://wi-cat.ru/forums/topic/dobavlenie-polzovateley-v-nag-connect/ . Non-authorized user can only see “topic unavailable” message. This page has counters for searching engine systems. So, for example, ya.ru (“Yandex”) is indexing this page, and search result we see is “No topic” as a content, but the following link https://wi-cat.ru/forums/topic/dobavlenie-polzovateley-v-nag-connect/ will be shown . In another words, topic names are shared as links to everybody, however its better for us that nobody should even knew about some topics existance . Another issue is if non-authorized user enters correct url of private topic, he can see that he has no access to is, however, using invalid url he only see that there is no such topic (it can also give an extra-information to a wrong person)
The only workaround is to add
<meta name="robots" content="noindex">
to all the hidden topics. Could you all this pls?
For example, we have a private topic https://wi-cat.ru/forums/topic/dobavlenie-polzovateley-v-nag-connect/ . Non-authorized user can only see “topic unavailable” message. This page has counters for searching engine systems. So, for example, ya.ru (“Yandex”) is indexing this page, and search result we see is “No topic” as a content, but the following link https://wi-cat.ru/forums/topic/dobavlenie-polzovateley-v-nag-connect/ will be shown . In another words, topic names are shared as links to everybody, however its better for us that nobody should even knew about some topics existance . Another issue is if non-authorized user enters correct url of private topic, he can see that he has no access to is, however, using invalid url he only see that there is no such topic (it can also give an extra-information to a wrong person)
The only workaround is to add
<meta name="robots" content="noindex">
to all the hidden topics. Could you all this pls?
Quote from Asgaros on December 14, 2018, 1:13 pmHello @rgn_sss
Yes, I get your point. I want to come with an alternative idea: How about redirecting an user back to the overview when he has no access to an area instead of showing an error-message?
I added this problem to my todo-list and will fix it in the upcoming v1.13 update which I plan to release in January:
https://github.com/Asgaros/asgaros-forum/issues/192
Hello @rgn_sss
Yes, I get your point. I want to come with an alternative idea: How about redirecting an user back to the overview when he has no access to an area instead of showing an error-message?
I added this problem to my todo-list and will fix it in the upcoming v1.13 update which I plan to release in January:
Quote from rgn_sss on December 14, 2018, 10:56 pmHello @asgaros , thanks for your idea, it solves an issue of getting the infomation about private topic existance, however the only “noindex” allows to exclude this link from searching results.
Thanks in advance for solving the problem in January!
Hello @asgaros , thanks for your idea, it solves an issue of getting the infomation about private topic existance, however the only “noindex” allows to exclude this link from searching results.
Thanks in advance for solving the problem in January!
Quote from Asgaros on December 17, 2018, 1:22 pm@rgn_sss
I am not 100% sure but if I redirect users for unaccessible topics directly back to the overview, the same should hold for search-engines?
Example – User has access to topic:
- mysite.com/forum/topic/secret works as expected
Example – User has no access to topic:
- User enters mysite.com/forum/topic/secret
- Server redirects user instantly to mysite.com/forum without showing anything (the browser-link will change as well)
In this case there should be no information-disclosure anymore because every randomly entered topic-name will redirects the user back to the overview as well. So there is no information for the user if there is a topic which does not exists or if there is an unaccessible topic.
I am not 100% sure but if I redirect users for unaccessible topics directly back to the overview, the same should hold for search-engines?
Example – User has access to topic:
- mysite.com/forum/topic/secret works as expected
Example – User has no access to topic:
- User enters mysite.com/forum/topic/secret
- Server redirects user instantly to mysite.com/forum without showing anything (the browser-link will change as well)
In this case there should be no information-disclosure anymore because every randomly entered topic-name will redirects the user back to the overview as well. So there is no information for the user if there is a topic which does not exists or if there is an unaccessible topic.
Quote from Asgaros on January 6, 2019, 8:28 pmHello again @rgn_sss
I just had a look at the problem again and the first mentioned approach with redirecting will not work as expected. Redirecting the user back to the overview when he cant access the topic is not a good solution because I think in some case an error-message is necessary. Imagine you want to see a topic but suddenly get redirected back to the overview. Does it happen because:
- you are not logged-in?
- the topic does not exist anymore?
- you lost access to an area?
So I will use the workaround you mentioned before: Adding nofollow-tags to the page when a link is not accessible without redirecting back and showing an error-message instead.
I think it is important to show a specific kind of error-message. Maybe a user has lost access to a topic because he is not logged-in anymore. If non-existing topics and unaccessible topics share the same error message, it would not be clear for him anymore why a topic seems to be unavailable.
So yeah, when an user somehow get a link to an unaccessible topic he can be aware of its meaning based on the URL. But those topics will not be available within the forum for him so the only way to get this topic-link is from an “insider”.
However: I will ensure that those kind of links will not get indexed anymore.
Hello again @rgn_sss
I just had a look at the problem again and the first mentioned approach with redirecting will not work as expected. Redirecting the user back to the overview when he cant access the topic is not a good solution because I think in some case an error-message is necessary. Imagine you want to see a topic but suddenly get redirected back to the overview. Does it happen because:
- you are not logged-in?
- the topic does not exist anymore?
- you lost access to an area?
So I will use the workaround you mentioned before: Adding nofollow-tags to the page when a link is not accessible without redirecting back and showing an error-message instead.
I think it is important to show a specific kind of error-message. Maybe a user has lost access to a topic because he is not logged-in anymore. If non-existing topics and unaccessible topics share the same error message, it would not be clear for him anymore why a topic seems to be unavailable.
So yeah, when an user somehow get a link to an unaccessible topic he can be aware of its meaning based on the URL. But those topics will not be available within the forum for him so the only way to get this topic-link is from an “insider”.
However: I will ensure that those kind of links will not get indexed anymore.
Quote from rgn_sss on January 7, 2019, 6:02 pmHello, @asgaros
sorry for my late response,
So I will use the workaround you mentioned before: Adding nofollow-tags to the page when a link is not accessible without redirecting back and showing an error-message instead.
thanks for this!
we also currently disabled feeds at our forum, ’cause it probably shared private topics as well ( when using https://our_web.ru/forums/topic/private_topic/?showfeed=rss2 )
Hello, @asgaros
sorry for my late response,
So I will use the workaround you mentioned before: Adding nofollow-tags to the page when a link is not accessible without redirecting back and showing an error-message instead.
thanks for this!
we also currently disabled feeds at our forum, ’cause it probably shared private topics as well ( when using https://our_web.ru/forums/topic/private_topic/?showfeed=rss2 )
Quote from Asgaros on January 7, 2019, 7:14 pmHello again @rgn_sss
Yes, they have also some meta-data included which have been removed from the latest development-version.
I plan to release v1.13 until the end of next week.
Hello again @rgn_sss
Yes, they have also some meta-data included which have been removed from the latest development-version.
I plan to release v1.13 until the end of next week.