User-agent: * Allow: /business/ Disallow: /*? Disallow: /sy Disallow: /us Disallow: /me # Twitter somehow uses their "bot" to make a "card" and thumbnail access have to be allowed User-agent: Twitterbot Allow: /business/ Disallow: /*? Disallow: /sy Disallow: /us Allow: /media/*.jpg$ Disallow: /me # NOTE: "Allow" and "Disallow" above are conflicting and the result is "undefined" according to Google. # As far as we tested, Twitter seems to take this as we intended.