https://observablehq.com/ logo
Join Slack
Powered by
Hi all, I am writing to request some help with a C...
# show-and-tell
f
Hi all, I am writing to request some help with a Collection of Maps that don't output greek characters correctly (example here: https://observablehq.com/@fotisz/greek-maps-and-administrative-boundaries-3-public?collection=@fotisz/greek-maps-and-administrative-boundaries). The issue happens in multiple notebooks in that collection and shows up like : ������ ���������� ����������, although these should be characters from the greek alphabet. Does anyone know if that encoding issue would be due to the shp/dbf files not having sufficient encoding declarations?
✅ 1
f
Hello, please make sure you share your notebook publicly?
f
Sure, can you check now?
a
It looks like they are encoded incorrectly in your source data. Do the properties display correctly for you locally?
f
They used to when I was calling them via an API = they no longer do since the API died and I had to upload them in each notebook
a
When I download your zip file and upload it in mapshaper, I get a message about it trying to auto detect the encoding. There it is rendering as Chinese characters. I am not behind a computer now to play more (using my phone), but I suspect you might be able to pre-process the file somewhere to get the encodings into something that is interpreted correctly
You can try this command outside a notebook: 
ogr2ogr -f "ESRI Shapefile" out_utf8.shp dhmosia_kthria.shp -lco ENCODING=UTF-8
f
It's actually greek characters, and if you put advanced options in mapshaper (encoding='UTF-8') then it doesn't auto detect - my Q here is how do I pick an encoding in Observable?
here is what I get in mapshaper - with the advanced encoding option to UTF-8 on
a
Copy code
async function shapefileZipToGeoJSON(zipBuffer, { encoding = "windows-1253" } = {}) {
  const files = fflate.unzipSync(zipBuffer instanceof Uint8Array ? zipBuffer : new Uint8Array(zipBuffer));
  const names = Object.keys(files);
  if (!names.length) throw new Error("Empty ZIP.");

  const shpName = names.find(n => n.toLowerCase().endsWith(".shp"));
  const dbfName = names.find(n => n.toLowerCase().endsWith(".dbf"));
  if (!shpName || !dbfName) throw new Error("ZIP must contain one .shp and one .dbf.");

  const toStream = (u8) => new Response(u8).body;
  const src = await shapefile.open(toStream(files[shpName]), toStream(files[dbfName]), { encoding });

  const features = [];
  for (;;) {
    const { done, value } = await src.read();
    if (done) break;
    features.push(value);
  }
  return { type: "FeatureCollection", features };
}
✅ 1
Try this. It should work. You need to require shapefile and fflate
f
Wow is this JS or Python (apologies a bit confused)
a
JavaScript
f
So I call the shapefileZipToGeoJSON instead of shp?
a
You also need shapefile = require("shapefile") And fflate = import("https://cdn.skypack.dev/fflate?min")
✅ 1
Yes, you do this: shapefileZipToGeoJSON(pBuildings)
You can name it whatever. I just made up something
f
Yeap, just tested - seems to be working thank you so much!
f
wow, thank you @aaronkyle!
a
You led me to fflate long ago, @Fil :) standing on the shoulders of giants!!
f
@Fil do you have something to add around the fflate technicalities?
f
nope!
f
So after some testing I think there might be an issue with the async - when applied to lat/lon there seems to be some rearrangement or change happening to lat/lon other than the character encodings. This is visible in this notebook: https://observablehq.com/@fotisz/gmab-hydrocounties?collection=@fotisz/greek-maps-and-administrative-boundaries where the tooltips appear at very random places or as if they have been rearranged
They definitely don't match what I would have expected them to be
f
The data is already projected on a planar reference, with coordinates in meters rather than in degrees of longitude and latitude. Instead of the "mercator" projection, you should use "reflect-y" (metric projections usually have y pointing to the North).
Copy code
Plot.plot({
  // Initialize the plot
  projection: { type: "reflect-y", domain: hydrocountiesEPSG4326GR }, // Set the projection
  width,
  marks: [
    Plot.geo(hydrocountiesEPSG4326GR, { fill: "lightblue", stroke: "darkblue" }), // Add the county boundaries
    Plot.geo(
      hydrocountiesEPSG4326GR,
      Plot.pointer(Plot.centroid({ stroke: "yellow" }))
    ),
    Plot.dot(hydrocountiesEPSG4326GR.features, Plot.centroid({fill: "red", stroke: "white"})),
    Plot.tip(
      hydrocountiesEPSG4326GR.features,
      Plot.pointer(Plot.centroid({ title: (d) => d.properties.rbdnamenl }))
    )
  ]
})
❤️ 1
f
I didn't have this problem previously with the Mercator Projection but will consider the update - there are issues in some other notebooks as well (of similar nature) ... should I try a similar solution there too?
f
Possibly? All the spherical projections in Plot expect coordinates to be expressed as degrees of lon/lat. If they are meters (or any other planar coordinate system) you should use 
"identity"
or 
"reflect-y"
, depending on whether y points down or up.
f
So when we say that a files is EPSG4326 doesn't that immediately imply Mercator projection?
f
EPSG4326 is lon, lat in degrees. You can use any spherical projection you want
but the file in your notebook doesn't appear to be following EPSG4326? I'm seeing coordinates in the range of 40000, which seem to indicate that they are already projected to a planar reference
f
Yeap, exactly!!
I think I had tried all other projections with the API but the Mercator was the only one working for all files that were EPSG 4326
Open in Slack
PreviousNext
Powered by