Compare commits
22 Commits
v1.0
...
developmen
| Author | SHA1 | Date | |
|---|---|---|---|
| fe8b033646 | |||
| 13c67bdad5 | |||
| 1e4b210670 | |||
| 4f3d8d7800 | |||
| 72b225bf10 | |||
| 5c536ccbe5 | |||
| d17b87c310 | |||
| ab997f2e22 | |||
| b52e31a4dc | |||
| 829407bb14 | |||
| cb429dd78e | |||
| b7de6d8119 | |||
| 65e352e8a4 | |||
| 38083d8a46 | |||
| ce676dea72 | |||
| 8548cc1f0f | |||
| 63202b53f1 | |||
| 2836a40616 | |||
| 19a8c2b997 | |||
| 217cd87feb | |||
| a0a023859a | |||
| 87f8ad4dae |
2
.gitignore
vendored
Normal file
2
.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
.venv/
|
||||
**/__pycache__/**
|
||||
4
.vscode/settings.json
vendored
Normal file
4
.vscode/settings.json
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"python.analysis.typeCheckingMode": "basic",
|
||||
"python.analysis.autoImportCompletions": true
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
# Guidlines
|
||||
|
||||
Code has to pass flake8 coding guidlines
|
||||
|
||||
http://flake8.pycqa.org/en/latest/
|
||||
[http://flake8.pycqa.org/en/latest/](http://flake8.pycqa.org/en/latest/)
|
||||
|
||||
78
README.md
78
README.md
@@ -4,16 +4,33 @@ Reverse GeoLocate from XMP sidecar files with optional LightRoom DB read
|
||||
|
||||
This script will update any of the Country Code, Country, State, City and Location data that is missing in sidecard files. If a Lightroom DB is set, it will read any data from the database and fill in the fields before it tries to get the location name from google with the Latitude and Longitude found in either the XMP sidecar file or the LR database.
|
||||
|
||||
#### Installing and setting up
|
||||
## Development Setup
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
.venv/bin/python -m pip install -U pip setuptools wheel
|
||||
pip freeze > requirements.txt
|
||||
```
|
||||
|
||||
```sh
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Then install the requests and python-xmp-toolkit modules from below
|
||||
|
||||
## Installing and setting up
|
||||
|
||||
The script uses the following external non defauly python libraries
|
||||
|
||||
* xmp toolkit
|
||||
* requests
|
||||
|
||||
install both with the pip3 command
|
||||
```
|
||||
pip3 install requests
|
||||
pip3 install python-xmp-toolkit
|
||||
|
||||
```sh
|
||||
pip install requests
|
||||
pip install python-xmp-toolkit
|
||||
```
|
||||
|
||||
XMP Toolkit also needs the [Exempi Library](http://libopenraw.freedesktop.org/wiki/Exempi). This one can be install via brew or macports directly.
|
||||
@@ -26,8 +43,9 @@ reverse_geolocate.py [-h] -i
|
||||
[-x [EXCLUDE XMP SOURCE FOLDER [EXCLUDE XMP SOURCE FOLDER ...]]]
|
||||
[-l LIGHTROOM FOLDER] [-s]
|
||||
[-f <overwrite, location, city, state, country, countrycode>]
|
||||
[-g GOOGLE API KEY] [-o] [-e EMIL ADDRESS] [-w]
|
||||
[-r] [-u] [-a] [-c] [-n] [-v] [--debug] [--test]
|
||||
[-d [FUZZY DISTANCE]] [-g GOOGLE API KEY] [-o]
|
||||
[-e EMIL ADDRESS] [-w] [-r] [-u] [-a] [-c] [-n]
|
||||
[-v] [--debug] [--test]
|
||||
|
||||
### Arguments
|
||||
|
||||
@@ -38,6 +56,7 @@ Argument | Argument Value | Description
|
||||
-l, --lightroom | Lightroom DB base folder | The folder where the .lrcat file is located. Optional, if this is set, LR values are read before any Google maps connection is done. Fills the Latitude and Longitude and the location names. Lightroom data never overwrites data already set in the XMP sidecar file. It is recommended to have Lightroom write the XMP sidecar file before this script is run
|
||||
-s, --strict | | Do strict check for Lightroom files and include the path into the check
|
||||
-f, --field | Keyword: overwrite, location, city, state, country, countrycode | In the default no data is overwritten if it is already set. With the 'overwrite' flag all data is set new from the Google Maps location data. Other arguments are each of the location fields and if set only this field will be set. This can be combined with the 'overwrite' flag to overwrite already set data
|
||||
-d, --fuzzy-cache | distance | Allow fuzzy cache lookup with either default value of 10m or an override value in m or km
|
||||
-n, --nobackup | | Do not create a backup of XMP sidecar file when it is changed
|
||||
-o, --openstreetmap | | Use OpenStreetMap instead of the default google maps
|
||||
-e, --email | email address | For OpenStreetMap with a large number of access
|
||||
@@ -51,25 +70,25 @@ Argument | Argument Value | Description
|
||||
--debug | | Full detailed debug output. Will print out alot of data
|
||||
--test | | Does not write any changed back to the XMP sidecar file. For testing purposes
|
||||
|
||||
The script will created a backup of the current sidecar file named <original name>.BK.xmp in the same location as the original file.
|
||||
The script will created a backup of the current sidecar file named `<original name>.BK.xmp` in the same location as the original file.
|
||||
|
||||
If the Lightroom lookup is used without the --strict argument and several files with the same name are found, they will be skipped for usage.
|
||||
|
||||
#### Example
|
||||
|
||||
```
|
||||
```sh
|
||||
reverse_geolocate.py -i Photos/2017/01 -i Photos/2017/02 -l LightRoom/MyCatalogue -f overwrite -g <API KEY>
|
||||
```
|
||||
|
||||
Will find all XMP sidecar files in both folders *Photos/2017/01* and *Photos/2017/02* and all folder below it. Uses the Lightroom database at *LightRoom/MyCatalogue*. The script will overwrite all data, even if it is already set
|
||||
|
||||
```
|
||||
```sh
|
||||
reverse_geolocate.py -i Photos/2017/01 -i Photos/2017/02 -x Photos/2017/02/Folder\ A -x Photos/2017/01/Folder\ B/some_file.xmp -l LightRoom/MyCatalogue
|
||||
```
|
||||
|
||||
Will exlucde "Photos/2017/02/Folder A" from processing. For -x also a file (including the .xmp extension) can be given.
|
||||
|
||||
```
|
||||
```sh
|
||||
reverse_geolocate.py -i Photos/2017/01/Event-01/some_photo.xmp -f location
|
||||
```
|
||||
|
||||
@@ -79,7 +98,7 @@ Only works on *some_photo.xmp* file and will only set the *location* field if it
|
||||
|
||||
The Google Maps API key and the OpenStreetMap Email address can be written to a config file with the -w argument. The config file is located in $HOME/.config/reverseGeolocate/reverse_geolocate.cfg in the following format
|
||||
|
||||
```
|
||||
```ini
|
||||
[API]
|
||||
googleapikey = <google api key>
|
||||
openstreetmapemail = <email>
|
||||
@@ -87,6 +106,12 @@ openstreetmapemail = <email>
|
||||
|
||||
if no -g or -e flag is given the keys are read from the config file. If the -g or -e parameter is given it will override the one found in the config file. A new parameter can be written to this config file with -w parameter.
|
||||
|
||||
### Cache lookups
|
||||
|
||||
If the same GPS coordinate is detected no other API maps call is done. With the fuzzy-distance argument this can be further extended to certain distances for each GPS coordinate from each other. The default value is 10m and can be overriden with an value to the argument.
|
||||
|
||||
Can be used to force cache on GPS coordinates that are very close to each other but not exactly the same.
|
||||
|
||||
### Google data priority
|
||||
|
||||
Based in the JSON return data the following fields are set in order. If one can not be found for a target set, the next one below is used
|
||||
@@ -122,22 +147,23 @@ order | type | target set
|
||||
|
||||
After the script is done the following overview will be printed
|
||||
|
||||
```
|
||||
=======================================
|
||||
XMP Files found : 57
|
||||
Updated : 3
|
||||
Skipped : 54
|
||||
New GeoLocation from Map : 2
|
||||
GeoLocation from Cache : 1
|
||||
Failed reverse GeoLocate : 0
|
||||
GeoLocaction from Lightroom : 1
|
||||
No Lightroom data found : 46
|
||||
More than one found in LR : 0
|
||||
```txt
|
||||
============================================
|
||||
XMP Files found : 57
|
||||
Updated : 3
|
||||
Skipped : 54
|
||||
New GeoLocation from Map : 2
|
||||
GeoLocation from Cache : 1
|
||||
GeoLocation from Fuzzy Cache : 0
|
||||
Failed reverse GeoLocate : 0
|
||||
GeoLocaction from Lightroom : 1
|
||||
No Lightroom data found : 46
|
||||
More than one found in Lightroom : 0
|
||||
```
|
||||
|
||||
If there are problems with getting data from the Google Maps API the complete errior sting will be printed
|
||||
|
||||
```
|
||||
```txt
|
||||
...
|
||||
---> Photos/2017/02/some_file.xmp: Error in request: OVER_QUERY_LIMIT You have exceeded your daily request quota for this API. We recommend registering for a key at the Google Developers Console: https://console.developers.google.com/apis/credentials?project=_
|
||||
(!) Could not geo loaction data [FAILED]
|
||||
@@ -146,13 +172,13 @@ If there are problems with getting data from the Google Maps API the complete er
|
||||
|
||||
Also the files that could not be updated will be printed at the end of the run under the stats list
|
||||
|
||||
```
|
||||
```txt
|
||||
...
|
||||
------------------------------
|
||||
----------------------------------------
|
||||
Files that failed to update:
|
||||
Photos/2017/02/some_file.xmp
|
||||
```
|
||||
|
||||
### Tested OS
|
||||
|
||||
This script has only been tested on macOS
|
||||
This script has only been tested on macOS
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
0
bin/utils/__init__.py
Normal file
0
bin/utils/__init__.py
Normal file
144
bin/utils/long_lat.py
Normal file
144
bin/utils/long_lat.py
Normal file
@@ -0,0 +1,144 @@
|
||||
"""
|
||||
latitude/longitude functions
|
||||
"""
|
||||
|
||||
import re
|
||||
from math import radians, sin, cos, atan2, sqrt
|
||||
|
||||
|
||||
def convert_lat_long_to_dms(lat_long, is_latitude=False, is_longitude=False):
|
||||
"""
|
||||
convert the LR format of N.N to the Exif GPS format
|
||||
|
||||
Args:
|
||||
lat_long(str): latLong in (-)N.N format
|
||||
is_latitude (bool, optional): flag, else we can't set North/Sout. Defaults to False.
|
||||
is_longitude (bool, optional): flag, else we can't set West/East. Defaults to False.
|
||||
|
||||
Returns:
|
||||
string: Deg,Min.Sec(NESW) format
|
||||
"""
|
||||
# minus part before . and then multiply rest by 60
|
||||
degree = int(abs(lat_long))
|
||||
minutes = round((float(abs(lat_long)) - int(abs(lat_long))) * 60, 10)
|
||||
if is_latitude is True:
|
||||
direction = "S" if int(lat_long) < 0 else "N"
|
||||
elif is_longitude is True:
|
||||
direction = "W" if int(lat_long) < 0 else "E"
|
||||
else:
|
||||
direction = "(INVALID)"
|
||||
return f"{degree},{minutes}{direction}"
|
||||
|
||||
|
||||
def convert_lat_to_dms(lat_long):
|
||||
"""
|
||||
wrapper functions for Long/Lat calls: latitude
|
||||
|
||||
Args:
|
||||
lat_long(str): latLong in (-)N.N format
|
||||
|
||||
Returns:
|
||||
string: Deg,Min.Sec(NESW) format
|
||||
"""
|
||||
return convert_lat_long_to_dms(lat_long, is_latitude=True)
|
||||
|
||||
|
||||
# wrapper for Long/Lat call: longitute
|
||||
def convert_long_to_dms(lat_long):
|
||||
"""
|
||||
wrapper for Long/Lat call: longitute
|
||||
|
||||
Args:
|
||||
lat_long(str): latLong in (-)N.N format
|
||||
|
||||
Returns:
|
||||
string: Deg,Min.Sec(NESW) format
|
||||
"""
|
||||
return convert_lat_long_to_dms(lat_long, is_longitude=True)
|
||||
|
||||
|
||||
def long_lat_reg(longitude, latitude):
|
||||
"""
|
||||
converts the XMP/EXIF formatted GPS Long/Lat coordinates
|
||||
from the <Degree>,<Minute.Second><NSEW> to the normal float
|
||||
number used in google/lr internal
|
||||
|
||||
Args:
|
||||
longitude(str): n,n.nNSEW format
|
||||
latitude(str): n,n.nNSEW format
|
||||
|
||||
Returns:
|
||||
dictionary: dict with converted lat/long
|
||||
"""
|
||||
# regex
|
||||
latlong_re = re.compile(r"^(\d+),(\d+\.\d+)([NESW]{1})$")
|
||||
# dict for loop
|
||||
lat_long = {"longitude": longitude, "latitude": latitude}
|
||||
# for element in lat_long:
|
||||
for index, element in lat_long.items():
|
||||
# match if it is exif GPS format
|
||||
_match = latlong_re.match(element)
|
||||
if _match is not None:
|
||||
# convert from Degree, Min.Sec into float format
|
||||
lat_long[index] = float(_match.group(1)) + (float(_match.group(2)) / 60)
|
||||
# if S or W => inverse to negative
|
||||
if _match.group(3) == "S" or _match.group(3) == "W":
|
||||
lat_long[index] *= -1
|
||||
return lat_long
|
||||
|
||||
|
||||
def convert_dms_to_lat(lat_long):
|
||||
"""
|
||||
rapper calls for DMS to Lat/Long: latitude
|
||||
|
||||
Args:
|
||||
lat_long(str): n,n.nNSEW format
|
||||
|
||||
Returns:
|
||||
dict: dict with converted lat/long
|
||||
"""
|
||||
return long_lat_reg("0,0.0N", lat_long)["latitude"]
|
||||
|
||||
|
||||
def convert_dms_to_long(lat_long):
|
||||
"""
|
||||
wrapper calls for DMS to Lat/Long: longitude
|
||||
|
||||
Args:
|
||||
lat_long(str): n,n.nNSEW format
|
||||
|
||||
Returns:
|
||||
dict: dict with converted lat/long
|
||||
"""
|
||||
return long_lat_reg(lat_long, "0,0.0N")["longitude"]
|
||||
|
||||
|
||||
def get_distance(from_longitude, from_latitude, to_longitude, to_latitude):
|
||||
"""
|
||||
calculates the difference between two coordinates
|
||||
|
||||
Args:
|
||||
from_longitude(str): from longitude
|
||||
from_latitude(str): from latitude
|
||||
to_longitude(str): to longitude
|
||||
to_latitude(str): to latitude
|
||||
|
||||
Returns:
|
||||
float: distance in meters
|
||||
"""
|
||||
# earth radius in meters
|
||||
earth_radius = 6378137.0
|
||||
# convert all from radians with pre convert DMS to long and to float
|
||||
from_longitude = radians(float(convert_dms_to_long(from_longitude)))
|
||||
from_latitude = radians(float(convert_dms_to_lat(from_latitude)))
|
||||
to_longitude = radians(float(convert_dms_to_long(to_longitude)))
|
||||
to_latitude = radians(float(convert_dms_to_lat(to_latitude)))
|
||||
# distance from - to
|
||||
distance_longitude = from_longitude - to_longitude
|
||||
distance_latitude = from_latitude - to_latitude
|
||||
# main distance calculation
|
||||
distance = (
|
||||
sin(distance_latitude / 2) ** 2 + cos(from_latitude) * cos(to_latitude) * sin(distance_longitude / 2) ** 2
|
||||
)
|
||||
distance = 2 * atan2(sqrt(distance), sqrt(1 - distance))
|
||||
return earth_radius * distance
|
||||
248
bin/utils/reverse_geolocate.py
Normal file
248
bin/utils/reverse_geolocate.py
Normal file
@@ -0,0 +1,248 @@
|
||||
"""
|
||||
reverse geolacte functions
|
||||
"""
|
||||
|
||||
import re
|
||||
import requests
|
||||
from utils.long_lat import long_lat_reg
|
||||
from utils.string_helpers import only_latin_chars
|
||||
|
||||
|
||||
def reverse_geolocate(longitude, latitude, map_type, args):
|
||||
"""
|
||||
wrapper to call to either the google or openstreetmap
|
||||
|
||||
Args:
|
||||
longitude (float): latitude
|
||||
latitude (float): longitue
|
||||
map_type(str): map search target (google or openstreetmap)
|
||||
args (_type_): _description_
|
||||
|
||||
Returns:
|
||||
_type_: dict with all data (see below)
|
||||
"""
|
||||
# clean up long/lat
|
||||
# they are stored with N/S/E/W if they come from an XMP
|
||||
# format: Deg,Min.Sec[NSEW]
|
||||
# NOTE: lat is N/S, long is E/W
|
||||
# detect and convert
|
||||
lat_long = long_lat_reg(longitude=longitude, latitude=latitude)
|
||||
# which service to use
|
||||
if map_type == "google":
|
||||
return reverse_geolocate_google(lat_long["longitude"], lat_long["latitude"], args)
|
||||
elif map_type == "openstreetmap":
|
||||
return reverse_geolocate_open_street_map(lat_long["longitude"], lat_long["latitude"], args)
|
||||
else:
|
||||
return {"Country": "", "status": "ERROR", "error": "Map type not valid"}
|
||||
|
||||
|
||||
def reverse_geolocate_init(longitude, latitude):
|
||||
"""
|
||||
inits the dictionary for return, and checks the lat/long on valid
|
||||
returns geolocation dict with status = 'ERROR' if an error occurded
|
||||
|
||||
Args:
|
||||
longitude (float): longitude
|
||||
latitude (float): latitude
|
||||
|
||||
Returns:
|
||||
_type_: empty geolocation dictionary, or error flag if lat/long is not valid
|
||||
"""
|
||||
# basic dict format
|
||||
geolocation = {
|
||||
"CountryCode": "",
|
||||
"Country": "",
|
||||
"State": "",
|
||||
"City": "",
|
||||
"Location": "",
|
||||
# below for error reports
|
||||
"status": "",
|
||||
"error_message": "",
|
||||
}
|
||||
# error if long/lat is not valid
|
||||
latlong_re = re.compile(r"^\d+\.\d+$")
|
||||
if not latlong_re.match(str(longitude)) or not latlong_re.match(str(latitude)):
|
||||
geolocation["status"] = "ERROR"
|
||||
geolocation["error_message"] = f"Latitude {latitude} or Longitude {longitude} are not valid"
|
||||
return geolocation
|
||||
|
||||
|
||||
def reverse_geolocate_open_street_map(longitude, latitude, args):
|
||||
"""
|
||||
OpenStreetMap reverse lookcation lookup
|
||||
|
||||
sample:
|
||||
https://nominatim.openstreetmap.org/reverse.php?format=jsonv2&
|
||||
at=<latitude>&lon=<longitude>&zoom=21&accept-languge=en-US,en&
|
||||
|
||||
Args:
|
||||
longitude (float): longitude
|
||||
latitude (float): latitude
|
||||
args (_type_): _description_
|
||||
|
||||
Returns:
|
||||
dictionary: dict with locaiton, city, state, country, country code
|
||||
if not fillable, entry is empty
|
||||
"""
|
||||
# init
|
||||
geolocation = reverse_geolocate_init(longitude, latitude)
|
||||
if geolocation["status"] == "ERROR":
|
||||
return geolocation
|
||||
# query format
|
||||
query_format = "jsonv2"
|
||||
# language to return (english)
|
||||
language = "en-US,en"
|
||||
# build query
|
||||
base = "https://nominatim.openstreetmap.org/reverse.php?"
|
||||
# parameters
|
||||
payload = {"format": query_format, "lat": latitude, "lon": longitude, "accept-language": language}
|
||||
# if we have an email, add it here
|
||||
if args.email:
|
||||
payload["email"] = args.email
|
||||
url = f"{base}"
|
||||
# timeout in seconds
|
||||
timeout = 60
|
||||
response = requests.get(url, params=payload, timeout=timeout)
|
||||
# debug output
|
||||
if args.debug:
|
||||
print(f"OpenStreetMap search for Lat: {latitude}, Long: {longitude}")
|
||||
if args.debug and args.verbose >= 1:
|
||||
print(f"OpenStreetMap response: {response} => JSON: {response.json()}")
|
||||
# type map
|
||||
# Country to Location and for each in order of priority
|
||||
type_map = {
|
||||
"CountryCode": ["country_code"],
|
||||
"Country": ["country"],
|
||||
"State": ["state"],
|
||||
"City": ["city", "city_district", "state_district"],
|
||||
"Location": ["county", "town", "suburb", "hamlet", "neighbourhood", "road"],
|
||||
}
|
||||
# if not error
|
||||
if "error" not in response.json():
|
||||
# get address block
|
||||
addr = response.json()["address"]
|
||||
# loop for locations
|
||||
for loc_index, sub_index in type_map.items():
|
||||
for index in sub_index:
|
||||
if index in addr and not geolocation[loc_index]:
|
||||
geolocation[loc_index] = addr[index]
|
||||
# for loc_index in type_map:
|
||||
# for index in type_map[loc_index]:
|
||||
# if index in addr and not geolocation[loc_index]:
|
||||
# geolocation[loc_index] = addr[index]
|
||||
else:
|
||||
geolocation["status"] = "ERROR"
|
||||
geolocation["error_message"] = response.json()["error"]
|
||||
print(f"Error in request: {geolocation['error']}")
|
||||
# return
|
||||
return geolocation
|
||||
|
||||
|
||||
def reverse_geolocate_google(longitude, latitude, args):
|
||||
"""
|
||||
Google Maps reverse location lookup
|
||||
|
||||
sample:
|
||||
http://maps.googleapis.com/maps/api/geocode/json?latlng=<latitude>,<longitude>&language=<lang>
|
||||
&sensor=false&key=<api key>
|
||||
|
||||
Args:
|
||||
longitude (float): longitude
|
||||
latitude (float): latitude
|
||||
args (_type_): _description_
|
||||
|
||||
Returns:
|
||||
dictionary: dict with location, city, state, country, country code
|
||||
if not fillable, entry is empty
|
||||
"""
|
||||
# init
|
||||
geolocation = reverse_geolocate_init(longitude, latitude)
|
||||
temp_geolocation = geolocation.copy()
|
||||
if geolocation["status"] == "ERROR":
|
||||
return geolocation
|
||||
# sensor (why?)
|
||||
sensor = "false"
|
||||
# language, so we get ascii en back
|
||||
language = "en"
|
||||
# request to google
|
||||
# if a google api key is used, the request has to be via https
|
||||
protocol = "https://" if args.google_api_key else "http://"
|
||||
base = "maps.googleapis.com/maps/api/geocode/json?"
|
||||
# build the base params
|
||||
payload = {"latlng": f"{latitude},{longitude}", "language": language, "sensor": sensor}
|
||||
# if we have a google api key, add it here
|
||||
if args.google_api_key:
|
||||
payload["key"] = args.google_api_key
|
||||
# build the full url and send it to google
|
||||
url = f"{protocol}{base}"
|
||||
# timeout in seconds
|
||||
timeout = 60
|
||||
response = requests.get(url, params=payload, timeout=timeout)
|
||||
# debug output
|
||||
if args.debug:
|
||||
print(f"Google search for Lat: {latitude}, Long: {longitude} with {response.url}")
|
||||
if args.debug and args.verbose >= 1:
|
||||
print(f"Google response: {response} => JSON: {response.json()}")
|
||||
# type map
|
||||
# For automated return of correct data into set to return
|
||||
type_map = {
|
||||
"CountryCode": ["country"],
|
||||
"Country": ["country"],
|
||||
"State": ["administrative_area_level_1", "administrative_area_level_2"],
|
||||
"City": ["locality", "administrative_area_level_3"],
|
||||
"Location": ["sublocality_level_1", "sublocality_level_2", "route"],
|
||||
}
|
||||
# print("Error: {}".format(response.json()['status']))
|
||||
if response.json()["status"] == "OK":
|
||||
# first entry for type = premise
|
||||
for entry in response.json()["results"]:
|
||||
for sub_entry in entry:
|
||||
if sub_entry == "types" and (
|
||||
"premise" in entry[sub_entry]
|
||||
or "route" in entry[sub_entry]
|
||||
or "street_address" in entry[sub_entry]
|
||||
or "sublocality" in entry[sub_entry]
|
||||
):
|
||||
# print("Entry {}: {}".format(sub_entry, entry[sub_entry]))
|
||||
# print("Address {}".format(entry['address_components']))
|
||||
# type
|
||||
# -> country,
|
||||
# -> administrative_area (1, 2),
|
||||
# -> locality,
|
||||
# -> sublocality (_level_1 or 2 first found, then route)
|
||||
# so we get the data in the correct order
|
||||
# for loc_index in type_map:
|
||||
# for index in type_map[loc_index]:
|
||||
for loc_index, sub_index in type_map.items():
|
||||
for index in sub_index:
|
||||
# this is an array, so we need to loop through each
|
||||
for addr in entry["address_components"]:
|
||||
# in types check that index is in there
|
||||
# and the location is not yet set
|
||||
# also check that entry is in LATIN based
|
||||
# NOTE: fallback if all are non LATIN?
|
||||
if index in addr["types"] and not geolocation[loc_index]:
|
||||
# for country code we need to use short name,
|
||||
# else we use long name
|
||||
if loc_index == "CountryCode":
|
||||
if only_latin_chars(addr["short_name"]):
|
||||
geolocation[loc_index] = addr["short_name"]
|
||||
elif not temp_geolocation[loc_index]:
|
||||
temp_geolocation[loc_index] = addr["short_name"]
|
||||
else:
|
||||
if only_latin_chars(addr["long_name"]):
|
||||
geolocation[loc_index] = addr["long_name"]
|
||||
elif not temp_geolocation[loc_index]:
|
||||
temp_geolocation[loc_index] = addr["long_name"]
|
||||
# check that all in geoloaction are filled and if not fille from temp_geolocation dictionary
|
||||
for loc_index in type_map:
|
||||
if not geolocation[loc_index] and temp_geolocation[loc_index]:
|
||||
geolocation[loc_index] = temp_geolocation[loc_index]
|
||||
# write OK status
|
||||
geolocation["status"] = response.json()["status"]
|
||||
else:
|
||||
geolocation["error_message"] = response.json()["error_message"]
|
||||
geolocation["status"] = response.json()["status"]
|
||||
print(f"Error in request: {geolocation['status']} {geolocation['error_message']}")
|
||||
# return
|
||||
return geolocation
|
||||
111
bin/utils/string_helpers.py
Normal file
111
bin/utils/string_helpers.py
Normal file
@@ -0,0 +1,111 @@
|
||||
"""
|
||||
various string helpers1
|
||||
"""
|
||||
|
||||
import unicodedata
|
||||
|
||||
# this is for looking up if string is non latin letters
|
||||
# this is used by isLatin and onlyLatinChars
|
||||
cache_latin_letters = {}
|
||||
|
||||
|
||||
def shorten_string(string, width, placeholder=".."):
|
||||
"""
|
||||
shortens a string to width and attached placeholder
|
||||
|
||||
Args:
|
||||
string(str): string to shorten
|
||||
width (int): length th shorten to
|
||||
placeholder (str, optional): optional string for removed shortend part. Defaults to '..'.
|
||||
|
||||
Returns:
|
||||
string: shortened string
|
||||
"""
|
||||
# get the length with double byte charactes
|
||||
string_length_cjk = string_len_cjk(str(string))
|
||||
# if double byte width is too big
|
||||
if string_length_cjk > width:
|
||||
# set current length and output string
|
||||
cur_len = 0
|
||||
out_string = ""
|
||||
# loop through each character
|
||||
for char in str(string):
|
||||
# set the current length if we add the character
|
||||
cur_len += 2 if unicodedata.east_asian_width(char) in "WF" else 1
|
||||
# if the new length is smaller than the output length to shorten too add the char
|
||||
if cur_len <= (width - len(placeholder)):
|
||||
out_string += char
|
||||
# return string with new width and placeholder
|
||||
return f"{out_string}{placeholder}"
|
||||
else:
|
||||
return str(string)
|
||||
|
||||
|
||||
def string_len_cjk(string):
|
||||
"""
|
||||
because len on string in python counts characters but we need the width
|
||||
count for formatting, we count two for a double byte characters
|
||||
|
||||
Args:
|
||||
string (string): string to check length
|
||||
|
||||
Returns:
|
||||
int: length including double count for double width characters
|
||||
"""
|
||||
# return string len including double count for double width characters
|
||||
return sum(1 + (unicodedata.east_asian_width(c) in "WF") for c in string)
|
||||
|
||||
|
||||
def is_latin(uchr):
|
||||
"""
|
||||
checks via the unciode class if a character is LATIN char based
|
||||
|
||||
from
|
||||
https://stackoverflow.com/a/3308844/7811993
|
||||
|
||||
Args:
|
||||
uchr (str): _description_
|
||||
|
||||
Returns:
|
||||
str: flagged LATIN or not char
|
||||
"""
|
||||
try:
|
||||
# if we found in the dictionary return
|
||||
return cache_latin_letters[uchr]
|
||||
except KeyError:
|
||||
# find LATIN in uncide type returned and set in dictionary for this character
|
||||
return cache_latin_letters.setdefault(uchr, "LATIN" in unicodedata.name(uchr))
|
||||
|
||||
|
||||
def only_latin_chars(unistr):
|
||||
"""
|
||||
chekcs if a string is based on LATIN chars. No for any CJK, Cyrillic, Hebrew, etc
|
||||
|
||||
from:
|
||||
https://stackoverflow.com/a/3308844/7811993
|
||||
|
||||
Args:
|
||||
unistr (str): string
|
||||
|
||||
Returns:
|
||||
bool: True/False for if string is LATIN char based
|
||||
"""
|
||||
return all(is_latin(uchr) for uchr in unistr if uchr.isalpha())
|
||||
|
||||
|
||||
def format_len(string, length):
|
||||
"""
|
||||
in case of CJK characters we need to adjust the format length dynamically
|
||||
calculate correct length based on string given
|
||||
|
||||
Args:
|
||||
string (str): string
|
||||
length (int): format length
|
||||
|
||||
Returns:
|
||||
int: adjusted format legnth
|
||||
"""
|
||||
# returns length udpated for string with double byte characters
|
||||
# get string length normal, get string length including double byte characters
|
||||
# then subtract that from the original length
|
||||
return length - (string_len_cjk(string) - len(string))
|
||||
13
pyproject.toml
Normal file
13
pyproject.toml
Normal file
@@ -0,0 +1,13 @@
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
[tool.black]
|
||||
# set 10 short for better formatting
|
||||
line-length = 110
|
||||
# to avoid " ... " " ... " string sets
|
||||
experimental-string-processing = true
|
||||
[tool.pylint.format]
|
||||
max-line-length = 120
|
||||
[tool.pylint.TYPECHECK]
|
||||
generated-members="utils.*"
|
||||
[tool.flake8]
|
||||
max-line-length = 120
|
||||
8
requirements.txt
Normal file
8
requirements.txt
Normal file
@@ -0,0 +1,8 @@
|
||||
certifi==2023.11.17
|
||||
charset-normalizer==3.3.2
|
||||
idna==3.6
|
||||
install==1.3.5
|
||||
python-xmp-toolkit==2.0.1
|
||||
pytz==2023.3.post1
|
||||
requests==2.31.0
|
||||
urllib3==2.1.0
|
||||
Reference in New Issue
Block a user