The Apache Haus Forum

Forum Topics => Third-party Modules => Topic started by: Gregg on August 28, 2009, 09:54:52 PM

Title: Seems mod_geoip2 was worth the effort
Post by: Gregg on August 28, 2009, 09:54:52 PM
Yes I admit at this time there are only 6 downloads of this module, for me it was worth it even if the count was at 0. Seeing use by others than I makes it that much better.

Yes, I'm bored :lol:
Title: Re: Seems mod_geoip2 was worth the effort
Post by: mario on August 30, 2009, 07:53:09 PM
For me it still works fine. On the company server it is funny to see that some customers come over a proxy in many countries.
Against your bornes: Write the tutorial how even I can compile apache  :P
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Sob on November 15, 2010, 02:30:16 AM
Current mod_geoip has one little disadvantage. It doesn't support IPv6. And not supporting IPv6 is bad, mmmkay... :)

I played with it a little and came with this:

http://web.hisoftware.cz/sob/download/mod_geoip2_1.2.5-ipv6-1.patch
http://web.hisoftware.cz/sob/download/GeoIP-1.4.6-win-ipv6-1.patch

When additional data files are added to Apache's config:

GeoIPDBFile "c:\path\to\your\GeoIPv6.dat"
GeoIPDBFile "c:\path\to\your\GeoLiteCity.dat"

Then it works fine for countries and does something for cities.

I have only one production IPv6 range to test with and it doesn't return any city info for it. For 6to4 it returns something, but complete nonsense. I tried to fake some addresses using GeoIPScanProxyHeaders and X-Forwarded-For and it shows something, but always somewhere in US, even for those where I'm sure they're somewhere else. So this part doesn't seem to work right. When looking at the csv version of GeoLiteCity database, it doesn't seem to include IPv6 at all. But in that case I don't get why it finds something.

You're welcome to join and play.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on November 23, 2010, 08:43:54 PM
Hi Sob,

"For 6to4 it returns something, but complete nonsense."

To be quite honest, I get complete nonsense at times in ipv4.

I have no outside IPv6, only internal so I cannot test this right now. However, I am working up a build for 2.3 that will build all modules (fcgid/macro/log_rotate/etc) in tree. As long as the mmn in ap_mmn.h keeps changing, modules have to be recompiled. Might as well do them during the compile of Apache.

When I get to geoip, I'll give these a try. If it doesn't hurt, I see no problem adding it to the bleeding edge alpha's collection of included modules. I'll talk it over with Mario also once I think I know what I am talking about  :D

Thanks for the patches.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Sob on November 24, 2010, 03:04:57 AM
Only GeoLiteCity.dat returns nonsense for IPv6 addresses. I did a little more reading since the last time and now I'm sure there's no IPv6 support in that database. Only available with IPv6 support is country database GeoIPv6.dat and it works correctly. So I removed the changes for city database. I also changed country part to avoid unnecessary lookups in wrong database:

http://web.hisoftware.cz/sob/download/mod_geoip2_1.2.5-ipv6-2.patch

It should be safe and correct. Except maybe for scanning proxy headers, I'm not sure if they would contain bare IPv6 address or enclosed in [ ].
Title: Re: Seems mod_geoip2 was worth the effort
Post by: tlitody on December 08, 2010, 02:33:16 PM
Thanks guys,

I just installed on mod_geoip win2K apache 2.2.11 and it seems to work fine.

Regarding city lookups. In my experience city lookup is a waste of time because the data is unreliable. The ISPs buy blocks of IP numbers and they all get assigned the town/city of the registered address of the ISP. When the ISPs allocate IP numbers to subscribers they don't assign town/city of the subscriber to the IP number. Also many ISPs use dynamically allocated IPs from their range so that one minute a visitor on one IP is in one city and an hour later the same IP may be being used by someone in another city.
The concept of using city lookups is fundamentally flawed due to inaccurate data. At least it is in the UK. Things may be different in different countries but I doubt it.
So I wouldn't waste your time on city and or lat/long modules.

You can check on the maxmind lookup at:

http://www.maxmind.com/app/locate_demo_ip

if I use my dedicated IP number in the UK it tells me I'm in Uxbridge about 180 miles from where I really am.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: ricardok1 on December 11, 2010, 02:05:44 AM
Hi there,

Anybody have a win32 vc6 compiled version of mod_geoip 1.4.6 or 1.4.7 beta ?

Thanks,
Ricardo
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on December 13, 2010, 09:12:33 PM
Quote from: tlitody on December 08, 2010, 02:33:16 PM
Regarding city lookups. In my experience city lookup is a waste of time because the data is unreliable. The ISPs buy blocks of IP numbers and they all get assigned the town/city of the registered address of the ISP. When the ISPs allocate IP numbers to subscribers they don't assign town/city of the subscriber to the IP number. Also many ISPs use dynamically allocated IPs from their range so that one minute a visitor on one IP is in one city and an hour later the same IP may be being used by someone in another city.
The concept of using city lookups is fundamentally flawed due to inaccurate data. At least it is in the UK. Things may be different in different countries but I doubt it.
So I wouldn't waste your time on city and or lat/long modules.

Well, Lat/Lon is totally useless unless you want the rough estimate of the lat/lon of whatever city.

I do know that although yes,city can be unreliable, I am finding to be getting better, and even when off it is getting close. I've been using this module for years now and just watching how it handled my own IP was telling enough.

Many years ago I was showing as coming from the city my ISP was located.
Then one day ISP switched the netblock pool I was being assigned from and it started showing a city that at least was in my county and 20 miles up the coast. Another IP some time later and it starte showing the city properly.

Then I moved, was assigned a IP from yet another net block and it was showing the correct city again. Then I got pushed to yet a different net block and now it is showing San Diego, which is not 100%, but close enough for nuclear weapons.

Obviously, anything free and mileage may vary, since I see ads targeted for my area on the likes of Youtube and others, I have a feeling the paid for database is rather accurate. Since they need to make a reason for you to pay, making the free database to be less accurate seems logical.

Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on December 13, 2010, 09:52:25 PM
OK, along with my diatribe above, I'm going to do this.

I finally remembered what I had to do to build this module in the first place so since that is the case, and this feature has now been asked for, I should have binaries up this week sometime.

Notice, they will be labeled with that horrible 12 letter word "Experimental," since it honestly is. I have no ipv6 outside the perimeter till we move into the 21st century here so cannot test the ipv6 lookups, however is seems to play nice on v4 which is good enough for me.

I have vc6 & vc9 x86 modules built at the moment, will do x64 when I crank up that box to build Apache 2.3.10. I may also be including it in the 2.3.10-alpha should I have time to work it into the build.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: mario on December 13, 2010, 11:17:43 PM
Quote from: Gregg on December 13, 2010, 09:52:25 PM
I have no ipv6 outside the perimeter till we move into the 21st century here so cannot test the ipv6 lookups, however is seems to play nice on v4 which is good enough for me.

With IPv6 you should be able to locate your toaster if linux with network is running on it  ;D
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on December 14, 2010, 12:09:39 AM
lmao ... via WiFi of course
Title: Re: Seems mod_geoip2 was worth the effort
Post by: tlitody on December 14, 2010, 09:43:23 AM
Quote from: Gregg on December 13, 2010, 09:12:33 PM
Well, Lat/Lon is totally useless unless you want the rough estimate of the lat/lon of whatever city.

I do know that although yes,city can be unreliable, I am finding to be getting better, and even when off it is getting close. I've been using this module for years now and just watching how it handled my own IP was telling enough.

My experience is in the UK where it just isn't anywhere near accurate. I think it really does depend on the ISPs updating the city/town of the end user. Or possibly some mechanism in the network to detect where the connection is coming is from but it certainly doesn't work in the UK.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: mario on December 14, 2010, 12:01:35 PM
In Germany it depends where the "point of presence" is (Hope that that is correct word german: Einwahlknoten) and the first DNS server in the backbone. Someday it is 150 km away and than only 20 km.
On the one hand it is bad not to locate users. On the other hand I don't want ppl to know exact where I live.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on December 14, 2010, 07:40:43 PM
Looking at the numbers, your toaster in Algeria Mario might easily show up a couple hundred miles away  from where it should :D

http://www.maxmind.com/app/geolite_city_accuracy
Title: Re: Seems mod_geoip2 was worth the effort
Post by: tlitody on December 15, 2010, 05:15:00 AM
Quote from: mario on December 14, 2010, 12:01:35 PM
In Germany it depends where the "point of presence" is (Hope that that is correct word german: Einwahlknoten) and the first DNS server in the backbone. Someday it is 150 km away and than only 20 km.
On the one hand it is bad not to locate users. On the other hand I don't want ppl to know exact where I live.

But even that would depend on the Point of presence having had its city attributed to a specific town/city.

I beleive the level of accuracy is really determined by the fact that the ISPs are huddled around internet backbone hubs which tend to be in big towns and cities with high population densities. Therefore there is a natural tendancy for a large part of the population to be living fairly close to their ISP. But once you get out of the big towns the level of accuracy will will be much lower.
I only raised it because it seems pointless (in the UK) for anyone to be trying to use city data to make any kind of business decisions. It's simply not accurate enough.

Title: Re: Seems mod_geoip2 was worth the effort
Post by: mario on April 16, 2011, 03:32:59 PM
I guess it got better! This is pretty close. Just the next town :)

Quote
Hello visitor from Buxtehude, Niedersachsen, Germany
IP Address: 91.97.163.111
Hostname: host-091-097-163-111.ewe-ip-backbone.de
Saturday, Apr 16, 2011 3:30 PM CEST
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on April 16, 2011, 11:43:40 PM
Getting there I guess. Maxmind is a US company, their real concern is the US and not the rest of the world. I'm sure they have to have contacts at ISPs to know how the ip blocks are assigned by them, so I am sure countries that are considered hostile to the US there is going to be a pretty poor return from  the DB.

It's interesting to note they do state a % of accuracy on the DBs. You'll notice even on the free one, the accuracy inside the US is pretty high, where outside the US it get's poor.  I'm sure it an accumulative average across all countries so because of the challenge noted above, it takes a hit on the % point. It's going to be interesting to see how they deal with IPv6 in the future.

Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on March 10, 2012, 11:55:42 AM
Hey Sob, I know you monitor this forum. Did you happen to see this?
http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Sob on March 19, 2012, 02:49:46 AM
No, I didn't notice it. And unfortunately is doesn't seem that I will be able to find time to play with it in near future. But on very quick look it seems that this might be enough:
diff -Naur mod_geoip2_1.2.5-ipv6-2a/mod_geoip.c mod_geoip2_1.2.5-ipv6-3/mod_geoip.c
--- mod_geoip2_1.2.5-ipv6-2a/mod_geoip.c 2010-11-24 02:35:01.726736800 +0100
+++ mod_geoip2_1.2.5-ipv6-3/mod_geoip.c 2012-03-19 02:44:47.935774500 +0100
@@ -500,7 +500,15 @@
break;
case GEOIP_CITY_EDITION_REV0:
case GEOIP_CITY_EDITION_REV1:
+ if(is_v6) break;
gir = GeoIP_record_by_addr(cfg->gips[i], ipaddr);
+ goto geoip_city_id;
+ case GEOIP_CITY_EDITION_REV0_V6:
+ case GEOIP_CITY_EDITION_REV1_V6:
+ if(!is_v6) break;
+ gir = GeoIP_record_by_addr_v6(cfg->gips[i], ipaddr);
+
+ geoip_city_id:
if (gir != NULL) {
        if ( gir->country_code != NULL ) {
  region_name = GeoIP_region_name_by_code(gir->country_code, gir->region);

It's completely untested, I just changed the file and didn't even try to compile it.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Sob on March 19, 2012, 03:06:34 AM
On second look, there is also mod_geoip2 1.2.7 and it seems to support everything. ;D
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on March 19, 2012, 09:05:51 PM
Yes, it seems to, tho I have not heard from anyone I have given it to with IPv6 yet. It requires the newer API 1.4.8 which I did get to build on Windows. I am waiting word on the IPv6 part and if it's working (even somewhat) will be putting it up here.

I have since done a r2 of it, due to how I dealt with the ssize_t in the API the first time is incorrect, even I have seen some odd returns with it, I redid it all and compiled again and so far I am not seeing someone I know to be in Australia showing as coming from AU, Iraq, Hungary and many others.
Title: Re: Seems mod_geoip2 was worth the effort
Post by: Gregg on March 06, 2013, 08:56:31 AM
1.2.8 source has been released. Haven't had the time to study it in depth yet. One thing I did see, it will ignore local IPs finally. This is a good thing since the module will crash at times when it happens to look up a local IP.