GUI terminal widget: Difference between revisions

From Octave
Jump to navigation Jump to search
(Insert content of https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00115.html.)
 
(→‎International Characters Support: Add one more bug to the list)
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Way back in January 2012, I posted the following message about changing
== Background ==
the terminal widget in the GUI to handle input differently.


Way back in January 2012, jwe [https://lists.gnu.org/archive/html/octave-maintainers/2012-01/msg00416.html posted the following message] about changing the terminal widget in the GUI to handle input differently.
This question was [https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00115.html addressed again in May 2019].


http://lists.gnu.org/archive/html/octave-maintainers/2012-01/msg00416.html
Jwe included a simple example written with gtkmm to illustrate the idea of having the terminal widget in the GUI be in control of input and output and feed lines of text to the interpreter.
To properly handle input that spans multiple lines, a push parser is used instead of the current pull parser.
It worked as an example, but was not much help for Octave since we are using Qt.
So now jwe has reworked it using Qt and you can find the sources here:


I included a simple example written with gtkmm to illustrate the idea of
https://hg.octave.org/jwe-qt-gui-with-push-parser
having the terminal widget in the GUI be in control of input and output
and feed lines of text to the interpreter.  To properly handle input
that spans multiple lines, a push parser is used instead of the current
pull parser.  It worked as an example, but was not much help for Octave
since we are using Qt.  So now I've reworked it using Qt and you can
find my sources here:


  http://hg.octave.org/jwe-qt-gui-with-push-parser
See the [https://hg.octave.org/jwe-qt-gui-with-push-parser/file/tip/NOTES NOTES file] for build instructions.
That file also contains the following list of open questions that will need to be resolved if we are going to attempt a switch.


See the NOTES file for build instructions.  That file also contains the
== Open questions ==
following list of open questions that will need to be resolved if we are
going to attempt a switch.


  * With this arrangement, would the interpreter have to run in a
=== Threads ===
separate thread?  As the example shows, it's not absolutely necessary.
It could offer some advantages, but only if it is possible for the GUI
to do useful things while the interpreter is busy.


  * If the interpreter is not running in a separate thread but but the  
# With this arrangement, would the interpreter have to run in a separate thread? As the example shows, it's not absolutely necessary.  It could offer some advantages, but only if it is possible for the GUI to do useful things while the interpreter is busy.
graphics engine is, then what happens to graphics callbacks when the  
# If the interpreter is not running in a separate thread but but the graphics engine is, then what happens to graphics callbacks when the parser is in the middle of parsing an expression?  Or is this not an issue because separate parsers can be used even if there is only one evaluator?  Could it ultimately be possible to have the evaluator running in multiple threads?
parser is in the middle of parsing an expression?  Or is this not an  
# If the interpreter does run in a separate thread, we still must wait for it to calculate and return a result so we can synchronize input and output.  Otherwise, we may print the next prompt before the output from the previous expression is evaluated and printed.  You'll see this behavior if you build the example program with CALC_USE_INTERPRETER_THREAD defined.
issue because separate parsers can be used even if there is only one  
evaluator?  Could it ultimately be possible to have the evaluator  
running in multiple threads?


  * If the interpreter does run in a separate thread, we still must
=== Multiple-line handling ===
wait for it to calculate and return a result so we can synchronize input
and output.  Otherwise, we may print the next prompt before the output
from the previous expression is evaluated and printed.  You'll see this
behavior if you build the example program with
CALC_USE_INTERPRETER_THREAD defined.


  * The example parser currently also performs evaluations and computes  
# The example parser currently also performs evaluations and computes results immediately so it doesn't properly handle expression lists that are split across multiple lines.  Octave wouldn't have this problem because we already build a parse tree then execute it once it is complete.
results immediately so it doesn't properly handle expression lists that  
# The example program doesn't attempt handle multi-line prompts or prompts with invisible characters (color specifications, for example). Fixing that will make the redisplay function significantly more complex. See, for example, how complicated the default rl_redisplay function is in the readline library.  Unless we actually write a terminal emulator (like the current terminal widgets) then it is not possible to use readline's rl_redisplay function directly.
are split across multiple lines.  Octave wouldn't have this problem  
because we already build a parse tree then execute it once it is complete.


  * Do we need text position markers to keep track of the prompt
=== Cursor position ===
position (beginning of current line) when inserting or clearing text?
This doesn't seem necessary in the current example, but it doesn't have
functions that can clear the screen or otherwise redraw prior output
that would cause the position of the cursor in the window to change.


  * The example program doesn't attempt handle multi-line prompts or  
# Do we need text position markers to keep track of the prompt position (beginning of current line) when inserting or clearing text? This doesn't seem necessary in the current example, but it doesn't have functions that can clear the screen or otherwise redraw prior output that would cause the position of the cursor in the window to change.
prompts with invisible characters (color specifications, for example).
Fixing that will make the redisplay function significantly more complex.
  See, for example, how complicated the default rl_redisplay function is
in the readline library.  Unless we actually write a terminal emulator
(like the current terminal widgets) then it is not possible to use
readline's rl_redisplay function directly.


  * We'll need to implement a pager ourselves, since "less" won't work
=== Pager ===
in this simplified terminal widget.


  * The system function may need to be modified so that external
# We'll need to implement a pager ourselves, since "less" won't work in this simplified terminal widget.
programs that expect to be running in a terminal will continue to work  
properly.  On Unixy systems, this job can be done with ptys.  I guess
Windows systems can use a hidden console?  But if these things are
required, are we more or less back to were we were before since we used
a pty and hidden console to implement the terminal widgets?  I believe
the Emacs start-process function must do similar things, so we might be
able to reuse that code.


  * If readline runs in the terminal widget, who owns the command-line
=== History ===
history?  Either way, if the GUI is in control of keyboard input, it
will need access to the history list and Octave will also need access
for the history functions.


Now that I have an almost working (if quite simplistic) example in Qt, I
# If readline runs in the terminal widget, who owns the command-line history?  Either way, if the GUI is in control of keyboard input, it will need access to the history list and Octave will also need access for the history functions.
will attempt to modify Octave to use this approachFrom that, I expect
 
many more questions to come up.
=== External programs / Portability ===
 
# The system function may need to be modified so that external programs that expect to be running in a terminal will continue to work properly.  On Unixy systems, this job can be done with ptysI guess Windows systems can use a hidden console?  But if these things are required, are we more or less back to were we were before since we used a pty and hidden console to implement the terminal widgets?  I believe the Emacs start-process function must do similar things, so we might be able to reuse that code.
 
 
=== [[International Characters Support]] ===
 
# Does GNU Readline support Unicode input?  What are the limiting factors to support Unicode?
#* '''MAYBE.''' If correctly configured, reasonably recent versions of GNU readline support 8bit characters if LC_CTYPE is set to a UTF-8 locale. Versions 6.1 and newer versions should be working if the user didn't override "meta-flag", "convert-meta", or "output-meta" (maybe also others). It is unclear from the documentation whether this also means support for all UTF-8 characters (including multibyte characters). But bash uses readline and it supports UTF-8. So it should be possible.
#* '''MAYBE.''' Setting LC_CTYPE to a UTF-8 locale is invalid on Windows but we could set the necessary flags manually.
#** Can we manage this problem in current MXE builds?
#*** '''MAYBE.''' See the patch in {{bug|47571}} ([https://savannah.gnu.org/bugs/?47571#comment42 comment #42]).
# Can the open bugs be resolved with the new solution?
#* {{bug|56173}} (probably unrelated)
#* {{bug|47571}} (probably resolved)
#* {{bug|43099}} (probably resolved)
#* {{bug|55689}} (probably resolved)
#* {{bug|50409}} (probably unrelated)
#* {{bug|59495}} (probably resolved)
#* {{bug|56224}} (probably resolved)
#* {{bug|53809}} (probably resolved)
#* {{bug|54069}} (probably resolved)
#* {{bug|58717}} (probably resolved)
 
=== Alternatives ===
 
# [https://github.com/jupyter/qtconsole qtconsole]
#* (+) Fancy interactive features including plots
#* (+) Contains history and a pager
#* (-) Written in Python
# [https://github.com/lxqt/qtermwidget qtermwidget]
#* (+) Reusable component (less Octave code) as shared and development library [https://packages.debian.org/stretch/libqtermwidget5-0 Debian],  [https://software.opensuse.org/package/libqtermwidget5-0 openSUSE].
#* (+) Contains history (and a pager?)
#* (+) Written in C++
#* (-) AFAIK, the current implementation was forked from this implementation some years ago. It deviated quite substantially which makes merging difficult and which might suggest that the original implementation doesn't fit our needs (confirmation needed).
#* (-) Not cross-platform ([https://github.com/lxqt/qtermwidget/issues/240 no Windows support]).
 
[[Category:Development]]

Revision as of 18:46, 1 December 2020

Background

Way back in January 2012, jwe posted the following message about changing the terminal widget in the GUI to handle input differently. This question was addressed again in May 2019.

Jwe included a simple example written with gtkmm to illustrate the idea of having the terminal widget in the GUI be in control of input and output and feed lines of text to the interpreter. To properly handle input that spans multiple lines, a push parser is used instead of the current pull parser. It worked as an example, but was not much help for Octave since we are using Qt. So now jwe has reworked it using Qt and you can find the sources here:

https://hg.octave.org/jwe-qt-gui-with-push-parser

See the NOTES file for build instructions. That file also contains the following list of open questions that will need to be resolved if we are going to attempt a switch.

Open questions

Threads

  1. With this arrangement, would the interpreter have to run in a separate thread? As the example shows, it's not absolutely necessary. It could offer some advantages, but only if it is possible for the GUI to do useful things while the interpreter is busy.
  2. If the interpreter is not running in a separate thread but but the graphics engine is, then what happens to graphics callbacks when the parser is in the middle of parsing an expression? Or is this not an issue because separate parsers can be used even if there is only one evaluator? Could it ultimately be possible to have the evaluator running in multiple threads?
  3. If the interpreter does run in a separate thread, we still must wait for it to calculate and return a result so we can synchronize input and output. Otherwise, we may print the next prompt before the output from the previous expression is evaluated and printed. You'll see this behavior if you build the example program with CALC_USE_INTERPRETER_THREAD defined.

Multiple-line handling

  1. The example parser currently also performs evaluations and computes results immediately so it doesn't properly handle expression lists that are split across multiple lines. Octave wouldn't have this problem because we already build a parse tree then execute it once it is complete.
  2. The example program doesn't attempt handle multi-line prompts or prompts with invisible characters (color specifications, for example). Fixing that will make the redisplay function significantly more complex. See, for example, how complicated the default rl_redisplay function is in the readline library. Unless we actually write a terminal emulator (like the current terminal widgets) then it is not possible to use readline's rl_redisplay function directly.

Cursor position

  1. Do we need text position markers to keep track of the prompt position (beginning of current line) when inserting or clearing text? This doesn't seem necessary in the current example, but it doesn't have functions that can clear the screen or otherwise redraw prior output that would cause the position of the cursor in the window to change.

Pager

  1. We'll need to implement a pager ourselves, since "less" won't work in this simplified terminal widget.

History

  1. If readline runs in the terminal widget, who owns the command-line history? Either way, if the GUI is in control of keyboard input, it will need access to the history list and Octave will also need access for the history functions.

External programs / Portability

  1. The system function may need to be modified so that external programs that expect to be running in a terminal will continue to work properly. On Unixy systems, this job can be done with ptys. I guess Windows systems can use a hidden console? But if these things are required, are we more or less back to were we were before since we used a pty and hidden console to implement the terminal widgets? I believe the Emacs start-process function must do similar things, so we might be able to reuse that code.


International Characters Support

  1. Does GNU Readline support Unicode input? What are the limiting factors to support Unicode?
    • MAYBE. If correctly configured, reasonably recent versions of GNU readline support 8bit characters if LC_CTYPE is set to a UTF-8 locale. Versions 6.1 and newer versions should be working if the user didn't override "meta-flag", "convert-meta", or "output-meta" (maybe also others). It is unclear from the documentation whether this also means support for all UTF-8 characters (including multibyte characters). But bash uses readline and it supports UTF-8. So it should be possible.
    • MAYBE. Setting LC_CTYPE to a UTF-8 locale is invalid on Windows but we could set the necessary flags manually.
      • Can we manage this problem in current MXE builds?
  2. Can the open bugs be resolved with the new solution?

Alternatives

  1. qtconsole
    • (+) Fancy interactive features including plots
    • (+) Contains history and a pager
    • (-) Written in Python
  2. qtermwidget
    • (+) Reusable component (less Octave code) as shared and development library Debian, openSUSE.
    • (+) Contains history (and a pager?)
    • (+) Written in C++
    • (-) AFAIK, the current implementation was forked from this implementation some years ago. It deviated quite substantially which makes merging difficult and which might suggest that the original implementation doesn't fit our needs (confirmation needed).
    • (-) Not cross-platform (no Windows support).