Fix Unicode positions

[dennwc]

Fix Unicode positions. Ideally this PR should also enable enable the token verifier, but it's not possible right now due to the way AST is structured. Some work in SDK is required to make it work.

Signed-off-by: Denys Smirnov [email protected]

---

This change is 

[creachadair]

Reviewed 8 of 8 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dennwc)

---

driver/fixtures/fixtures_test.go, line 37 at r1 (raw file):

		},
	},
	// TODO: This won't really work because of the C++ macros expansion.

I don't quite follow: Of course macro expansions affect position, but if the AST we generate doesn't expand the macro anyway aren't the locations correct?

[dennwc]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @creachadair)

---

driver/fixtures/fixtures_test.go, line 37 at r1 (raw file):

Previously, creachadair (M. J. Fromberger) wrote…

I don't quite follow: Of course macro expansions affect position, but if the AST we generate doesn't expand the macro anyway aren't the locations correct?

Turned out it does expand macros that are in the same file. It sill preserves the AST structure for it but breaks the positional info that we use.

To give an example, there was a macro that joins two values, so the source line is JOIN(3, 4). But the preprocessor actually replaces it with a string 34, so the token verification gets confused and can't verify it based on the source file.

So maybe other positional info is also incorrect for files that use macros.

[creachadair]

Reviewable status: complete! all files reviewed, all discussions resolved

---

driver/fixtures/fixtures_test.go, line 37 at r1 (raw file):

Previously, dennwc (Denys Smirnov) wrote…

Turned out it does expand macros that are in the same file. It sill preserves the AST structure for it but breaks the positional info that we use.

To give an example, there was a macro that joins two values, so the source line is JOIN(3, 4). But the preprocessor actually replaces it with a string 34, so the token verification gets confused and can't verify it based on the source file.

So maybe other positional info is also incorrect for files that use macros.

Urk, that's a little scary. I wonder if we can disable macro expansion entirely rather than have it depend on where the definition resides. Not a topic for this PR, though.

[dennwc]

Reviewable status: complete! all files reviewed, all discussions resolved

---

driver/fixtures/fixtures_test.go, line 37 at r1 (raw file):

Previously, creachadair (M. J. Fromberger) wrote…

Urk, that's a little scary. I wonder if we can disable macro expansion entirely rather than have it depend on where the definition resides. Not a topic for this PR, though.

Opened an issue for it: #41