[Dvipng] Re: [AUCTeX-devel] preview-latex, dvipng, and LyX

[Angus Leeming]

However, in recent versions of preview-latex and dvipng the output
$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 (preview-latex version 11.81) depth=6 height=16]
this output confuses the script LyX uses for generating preview

A lesson is learnt, but the damage is irreversible.
I mean, dvipng and preview.sty have been released in this manner, so
there is no sane around letting LyX detect this string and deal with
it. That's the only way to minimize the impact, even if dvipng get
changed again in the next version to give this information in a
they will have to be a part of damage control.

Hi, David!

Damage control from this side is easy: no version of LyX has been released
that uses dvipng. (That will be part of the upcoming 1.4.0 release.) That
means we can jump through whatever hoops you choose to present us with ;-)

The obvious solution is to add a flag to dvipng
--LyX
that is used to swap the existing BE_NONQUIET for BE_VERBOSE

A second problem that I noticed is that when a font is missing and
mktexpk (or alike) gets called to generate it, its stdout gets
intermixed with that of dvipng, causing a similar (but worse) confusion.
Is it possible to have the stdout of those helper programs redirected to
stderr?

The latter would actually be useful for preview-latex as well, only
that it would be nice to direct the output completely elsewhere (Emacs
does not capture stderr separately): I often have to run preview-latex
several times when new fonts get generated, because the font
generating output confuses it.

Perhaps one reasonable solution would be to add a flag to dvipng
--redirect_externals_stdout_to_stderr

Or somesuch ;-)

[Enrico Forestieri]

Hello,

sorry if this is not the right list. In this case, please, redirect me
to the right one.

As you know, LyX (http://www.lyx.org) uses both the preview-latex package
and dvipng to display snippet previews of LaTeX math formulas. In order
to do so it also relies on the output (stdout) of dvipng to capture the
snippet number and its height and depth.

However, in recent versions of preview-latex and dvipng the output
devised for LyX's sake gets corrupted in the following way:

$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 (preview-latex version 11.81) depth=6 height=16]

this output confuses the script LyX uses for generating preview
snippets, which, instead, is expecting an output like this:

$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 depth=6 height=16]

I found that the problem is due to both recent versions of preview-latex
and dvipng. For example, this problem doesn't show up in debian testing
but manifests itself on Windows with MikTeX (where a more recent version
of preview.sty is installed).

Here is the relevant part of the diff between the versions of preview.sty
in debian tetex and in MikTeX:

$ diff -u preview.sty.tetex preview.sty.miktex
...
@@ -83,6 +85,7 @@
\DeclareOption{dvips}{%
\let\***@graphicstype\@ne
\***@delay{\AtBeginDvi{%
+ \special{!/***@version(\***@version)def}
\special{!userdict begin/preview-bop-level 0 def%
/bop-hook{/preview-bop-level dup load dup 0 le{/isls false def%
/vsize 792 def/hsize 612 def}if 1 add store}bind def%
...


and here is the relevant code in the dvipng (version 1.7) special.c source
file causing the problem:


if (strncmp(buffer,"!/***@version(",18)==0) {
buffer+=18;
length-=18;
while (length>0 && buffer[length]!=')')
length--;
if (page_imagep==NULL)
Message(BE_NONQUIET," (preview-latex version %.*s)",length,buffer);
return;
}

/* preview-latex' tightpage option */
if (strncmp(buffer,"!/***@tightpage",19)==0) {
buffer+=19;
SKIPSPACES(buffer);
if (strncmp(buffer,"true",4)==0) {
if (page_imagep==NULL)
Message(BE_NONQUIET," (preview-latex tightpage option detected,\
will use its bounding box)");
flags |= PREVIEW_LATEX_TIGHTPAGE;
return;
}
}
if (strncmp(buffer,"!userdict",9)==0
&& strstr(buffer+10,"7{currentfile token not{stop}if 65781.76 div")!=NULL)
{
if (page_imagep==NULL && ~flags & PREVIEW_LATEX_TIGHTPAGE)
Message(BE_NONQUIET," (preview-latex <= 0.9.1 tightpage option detected,\
will use its bounding box)");
flags |= PREVIEW_LATEX_TIGHTPAGE;
return;
}


As a solution, it would suffice to replace BE_NONQUIET with
BE_VERBOSE in the code snippet above, such as to not clutter stdout
between "[1 " and "depth=...]".

I would like to ask if this acceptable for you, or, in case it is not,
if you can suggest an alternative. In this regard, what it is important
is that nothing goes inside "[" ... "]" except snippet number and depth
and height information.

A second problem that I noticed is that when a font is missing and
mktexpk (or alike) gets called to generate it, its stdout gets intermixed
with that of dvipng, causing a similar (but worse) confusion.

Is it possible to have the stdout of those helper programs redirected to
stderr?

Thank you for your attention.

--
Enrico

[David Kastrup]

Post by Enrico Forestieri
sorry if this is not the right list. In this case, please, redirect me
to the right one.

The list is quite correct, though there might be a dvipng-specific
list, too.

Post by Enrico Forestieri
As you know, LyX (http://www.lyx.org) uses both the preview-latex
package and dvipng to display snippet previews of LaTeX math
formulas. In order to do so it also relies on the output (stdout) of
dvipng to capture the snippet number and its height and depth.
However, in recent versions of preview-latex and dvipng the output
$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 (preview-latex version 11.81) depth=6 height=16]
this output confuses the script LyX uses for generating preview

A lesson is learnt, but the damage is irreversible.

I mean, dvipng and preview.sty have been released in this manner, so
there is no sane around letting LyX detect this string and deal with
it. That's the only way to minimize the impact, even if dvipng get
changed again in the next version to give this information in a
different way. So I am copying the LyX developer list with this mail:
they will have to be a part of damage control.

Post by Enrico Forestieri
$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 depth=6 height=16]
I found that the problem is due to both recent versions of preview-latex
and dvipng. For example, this problem doesn't show up in debian testing
but manifests itself on Windows with MikTeX (where a more recent version
of preview.sty is installed).
Here is the relevant part of the diff between the versions of preview.sty
$ diff -u preview.sty.tetex preview.sty.miktex
...
@@ -83,6 +85,7 @@
\DeclareOption{dvips}{%
\special{!userdict begin/preview-bop-level 0 def%
/bop-hook{/preview-bop-level dup load dup 0 le{/isls false def%
/vsize 792 def/hsize 612 def}if 1 add store}bind def%
...
and here is the relevant code in the dvipng (version 1.7) special.c source
buffer+=18;
length-=18;
while (length>0 && buffer[length]!=')')
length--;
if (page_imagep==NULL)
Message(BE_NONQUIET," (preview-latex version %.*s)",length,buffer);
return;
}
/* preview-latex' tightpage option */
buffer+=19;
SKIPSPACES(buffer);
if (strncmp(buffer,"true",4)==0) {
if (page_imagep==NULL)
Message(BE_NONQUIET," (preview-latex tightpage option detected,\
will use its bounding box)");
flags |= PREVIEW_LATEX_TIGHTPAGE;
return;
}
}
if (strncmp(buffer,"!userdict",9)==0
&& strstr(buffer+10,"7{currentfile token not{stop}if 65781.76 div")!=NULL)
{
if (page_imagep==NULL && ~flags & PREVIEW_LATEX_TIGHTPAGE)
Message(BE_NONQUIET," (preview-latex <= 0.9.1 tightpage option detected,\
will use its bounding box)");
flags |= PREVIEW_LATEX_TIGHTPAGE;
return;
}
As a solution, it would suffice to replace BE_NONQUIET with
BE_VERBOSE in the code snippet above, such as to not clutter stdout
between "[1 " and "depth=...]".
I would like to ask if this acceptable for you, or, in case it is not,
if you can suggest an alternative. In this regard, what it is important
is that nothing goes inside "[" ... "]" except snippet number and depth
and height information.
A second problem that I noticed is that when a font is missing and
mktexpk (or alike) gets called to generate it, its stdout gets intermixed
with that of dvipng, causing a similar (but worse) confusion.
Is it possible to have the stdout of those helper programs redirected to
stderr?

The latter would actually be useful for preview-latex as well, only
that it would be nice to direct the output completely elsewhere (Emacs
does not capture stderr separately): I often have to run preview-latex
several times when new fonts get generated, because the font
generating output confuses it.

[Jan-Åke Larsson]

However, in recent versions of preview-latex and dvipng the output
$ dvipng ...
This is dvipng 1.7 Copyright 2002-2005 Jan-Ake Larsson
[1 (preview-latex version 11.81) depth=6 height=16]
this output confuses the script LyX uses for generating preview

A lesson is learnt, but the damage is irreversible.

Well, dvipng has always been chattering about things it does on each
page, even though this particular message will always appear on page 1
from now on. For instance, if the dvi page numbers differ from the
physical page numbers, there will be a heads-up in the form of
"(physical page number)". And if an eps is included you'll see
"<foo.eps>".

All these messages can't be put on stderr because they do belong on
certain pages. I wonder if the lyx people would be so kind as to look
for just the "depth=" and "height=" strings, these will not change
over time. I suppose that

"\[([0-9]+) .*?depth=(-?[0-9]+) height=(-?[0-9]+)"

would do the trick. Or is there some deep reason for not using ".*?" ?

A second problem that I noticed is that when a font is missing and
mktexpk (or alike) gets called to generate it, its stdout gets
intermixed with that of dvipng, causing a similar (but worse) confusion.
Is it possible to have the stdout of those helper programs redirected to
stderr?

The latter would actually be useful for preview-latex as well, only
that it would be nice to direct the output completely elsewhere (Emacs
does not capture stderr separately): I often have to run preview-latex
several times when new fonts get generated, because the font
generating output confuses it.

That pattern would get rid of this problem as well: IIRC the output of
the mktexpk script is always a closed "[1] [14] ..." in which case the
"\[([0-9]+) " part would only match BOP because of the trailing space
and the "height=(-?[0-9]+)" would match EOP.

Anyhow, the font-generating call is internal to kpathsea, it is not as
if I get to choose where to send output. With glibc redirection should
be easy, but without glibc I think I'll need to fork for every call to
kpse_find_pk, and redirect or gobble output and _then_ call kpse_find_pk
_again_ in the parent. Hmmmmmmmmm. Some overhead, but possible.

Opinions?
/JÅ

[Jan-Åke Larsson]

Post by Jan-Åke Larsson
"\[([0-9]+) .*?depth=(-?[0-9]+) height=(-?[0-9]+)"

Okay, I haven't been as diligent as I should. There may be a starting
paranthesis instead of a space after the page number. A working pattern
is

metrics_re = re.compile(\
"\[([0-9]+)[ \[\(<].*?depth=(-?[0-9]+) height=(-?[0-9]+)" \
)

/JÅ

[Jan-Åke Larsson]

Do you really mean "[ \[\(<].*?depth"? Translates as
a single space, [, ( or < character followed by (zero or more "any
characters") zero or one times followed by "depth".
I don't think so...

I do. What I meant was I havent (until now) made sure there is a space
after the page number. So to make this work with present version(s) of
dvipng I had to come up with something.

You mean mktexpk and newlines, right? (gs won't get to print anything on
stdout.) So I'm thinking about the possibility of outputting the page
number just before the "depth" text. Problem is, the page number is
output before the initial run is made, the run that calculates the size
of the image. And if a font generation occurs, I really really need that
to know the sizes of the glyphs. I could output "[page" after the
initial run but then I'd lose the nice structure of things needed in the
page being inside the [].

Oh, bugger. What about a simple but perhaps crude way out: a --page
switch, that will make the thing output "page=23414" just before
"depth=12"?

Downside: things will stop working with earlier versions because this
will be interpreted as -p0 : "start at page 0" and there is no page 0 in
most DVIs, so no output. Bah.

Perhaps I should just output "page=" as soon as --depth or --height is
in effect.

/JÅ

[Jan-Åke Larsson]

Post by Jan-Åke Larsson
Perhaps I should just output "page=" as soon as --depth or --height is
in effect.

Or "preview=" as soon as the preview package is detected?
/JÅ

[Jan-Åke Larsson]

Either/any. Something simple that doesn't make life difficult for you and
allows us to parse your messages robustly.

What about the following patch?

a) Page numbers from dvipng always match "\[[\d]+"
b) Glyph numbers from mktexpk always match "\[[\d]+\]".
c) No other numbers in dvipng's output will match "\[[\d]+"
d) Since you use --depth, page numbers from dvipng will always match
"\[[\d]+[^]]", that is, will not be followed by "\]".

The provided patch searches _first_ for "\[([\d]+)[^]]" and _then_ for
"depth=([\d]+) height=([\d]+)". Bonus: will work with any previous
version of dvipng.

But your point is that another (future) helper called from within dvipng
might not heed this, or?

/JÅ

[Jan-Åke Larsson]

Either/any. Something simple that doesn't make life difficult for you and
allows us to parse your messages robustly.

What about the following patch?

a) Page numbers from dvipng always match "\[[\d]+"
b) Glyph numbers from mktexpk always match "\[[\d]+\]".
c) No other numbers in dvipng's output will match "\[[\d]+"
d) Since you use --depth, page numbers from dvipng will always match
"\[[\d]+[^]]", that is, will not be followed by "\]".

The provided patch searches _first_ for "\[([\d]+)[^]]" and _then_ for
"depth=([\d]+) height=([\d]+)". Bonus: will work with any previous
version of dvipng.

But your point is that another (future) helper called from within dvipng
might not heed this, or?

/JÅ

[Jan-Åke Larsson]

Post by Jan-Åke Larsson
What about the following patch?

[Enrico Forestieri]

Post by Jan-Åke Larsson
What about the following patch?

Whooo! I didn't expect you to do our work for us. Thanks!
In answer: I like the idea. Enrico, could you try it out to see if it cures
your problems?

At last I found this post on gmane... it did not appear in the listing and
I found that I had to click on the subject line to access the whole thread.

Yes, the patch works.

--
Enrico

[Jan-Åke Larsson]

What about freopen? I tested the code below under solaris, linux and
windows with both mingw (gcc -mno-cygwin, really) and the
free MSVC compiler. All tests succeeded!

Ah, yes, that would be a possibility. freopen, dup and dup2 is
reasonably portable in the UNIX world, although the ones you tested are
usually counted as fairly standard systems. dvipng (as distributed in
tetex) needs to build on other Unixes, for example IRIX, although I
think there won't be much problem with these particular functions; I
already use dup2 for the call to gs. Unfortunately my only way to test
this would be to release and wait for build failures to drop in (or use
an autoconf test).

Also I don't know how to do this in the MIKTeX environment, and the real
problem appears mostly there: in any fairly recent UNIX (as the ones you
tested) freetype or t1lib is available so there mktexpk never gets
called. Probably it is relatively easy to port it, but I'd need to rely
on Christian Schenk to do that.

Perhaps it won't be necessary at all, we'll see. But thanks for the pointer.

/JÅ