UAE and the mysteries of unreachable websites

0plus1 asked:

I write here because I’m really lost, please stay with me because it’s not easy to explain.

A company asked me to set-up a private server, now I’m a programmer so I got a solution with technical support and cpanel which helped me to setup everything and it’s working smoothless. I’m by no means a professional sysadmin, but I have a fair knowledge of server configurations, but this problem is way over my knowledge, and apparently way over the knowledge of most sysadmins, I really hope that here I’ll find someone with enough experience to help me or at least give me more insight.

Now this company for which I’m consulting operates in the UAE (United Arab Emirates) and from there the server is almost unreachable. It started with ns not registering in the UAE, after a week that sorted itself out and now the site is indeed reachable, but it takes almost 2 minutes to load a webpage with one line of text. Emails go in timeout.

The domain currently parked there has been bought appositely for tests, the main one that was supposed to go there, after a catastrophic week has been transferred to a shared hosting solution in the UK, and from there it works like a charme.

Now after doing some research I discovered that I’m not alone in this, there are several reports of webmasters discovering that their website is not reachable inside the UAE, and mind this has nothing to do with the state-wide block of questionable sites, because in that case an error message appears, this seems to be related to the infrastructure of the UAE, which apparently reroutes everything through their own “fake” internet.

Apparently new servers with their own IP are not recognized (yet?) by the UAE infrastructure, while shared hosting solutions seeing that they operates tons of other websites are more likely to be part of the UAE network.

Now my questions are:

1) Has someone a real explanation for this? The only thing I can think of is that the server is on a new IP that is not yet recognized by the UAE, but that doesn’t explain why it loads (even if after 2 minutes). I don’t have any help from within the UAE as the only people that are “experts” are questionable companies that simply try to sell their own services.

2) If there is really some kind of block of new servers, is it possible to know before if a server is reachable from within the UAE, currently this is not a ns problem as even accessing the server with its IP result in a 2 minute wait.

3) Can it be that the problem lies somewhere else? There are some tests that I can perform? I’m not physically in the UAE, but I can ask the people there, or use teamviewer. Could it be some misconfiguration on the server (mind that the site works EVERYWHERE else in the world).

Thank you for ANY kind of help

My answer:

Here is what’s going on:

The two major ISPs in UAE, Etisalat and du, use SmartFilter by Secure Computing (now McAfee) to block content. This works by transparently proxying all users’ web traffic through the filters.

What happens then: When a user requests a web site, the SmartFilter checks its local blacklist (for instance, in UAE, all sites ending in .il are blacklisted) then looks up its classification (e.g. politics, news, social media, games, etc.) to see if it’s appropriate.

What happens is that new sites which have never been seen before are unclassified until someone at McAfee visits and classifies the site. This can take a few days. Companies and ISPs which use tools like SmartFilter can choose to allow or block unclassified sites, and it seems that here, they are blocked.

The other problem with proxies set up in this manner is that they must interfere with the end-to-end connection between a user and the remote site. This can and sometimes does result in performance problems which are difficult to resolve. You may have a lot of difficulty finding the responsible person at the ISP who can actually help to resolve these sorts of issues.

Services other than web sites (HTTP) can be interfered with in other ways, as well.

To resolve the issue, set up a VPN between your corporate network in UAE and a server in another country without censorship, the sole purpose of which is to carry your company Internet traffic.

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.