Fix visual artefacts with partial L2 instancing

Closes #3634

To produce inferred deltas that will be correct given OpenType's gvar
semantics, fontTool's IUP optimisation module checks the equality of
some points. However, this happens before the points are rounded,
whereas the point comparison that happens at runtime will occur after
the points are rounded (as is necessary to serialise glyf), which leads
to diverging semantics and so diverging and incorrect implied deltas.

This leads to significant visual artefacts, e.g. where large deltas that
should be inferred based on previous values are instead interpreted as 0
at runtime.

I suspect this has gone undetected as the subsetter normally works with
rounded points; in the rarer case that partial VF instancing is
occurring with a different default position, however, varLib.instancer
will calculate and apply the relevant deltas to the font's original
coordinates to effect the new default position, which leads to unrounded
points in memory. This commit ensures that we round directly before
optimising (but still after calculating `glyf` metrics, for backward
compatibility).

Lib/fontTools/varLib/instancer/__init__.py

@@ -897,7 +897,18 @@ def _instantiateGvarGlyph(
         return
 
     if optimize:
+        # IUP semantics depend on point equality, and so round prior to
+        # optimization to ensure that comparisons that happen now will be the
+        # same as those that happen at render time. This is especially needed
+        # when floating point deltas have been applied to the default position.
+        #     See https://github.com/fonttools/fonttools/issues/3634
+        # Rounding must happen only after calculating glyf metrics above, to
+        # preserve backwards compatibility.
+        #     See 0010a3cd9aa25f84a3a6250dafb119743d32aa40
+        coordinates.toInt()
+
         isComposite = glyf[glyphname].isComposite()
+
         for var in tupleVarStore:
             var.optimize(coordinates, endPts, isComposite=isComposite)