You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
(py/run-simple-string "from tqdm import tqdm")
(py/run-simple-string "from time import sleep")
(def s "with tqdm(total=100) as pbar:\n for i in range(10):\n sleep(1)\n pbar.update(10)")
(py/run-simple-string s)
This code should print something (in realtime(, so every second for 10 seconds,
this is behavoiur in teh python interpreter.
But it does not in current `libpython-clj.
I can "trigger" the printing manually, by "doing something" after th above lines, which make Clojure print someting to "err" or "flush it".
(it will not print every second, but all togerther at the end)
(require '[libpython-clj2.python :as py])
(py/initialize!)
(def _ (py/run-simple-string "from tqdm import tqdm"))
(def _ (py/run-simple-string "from time import sleep"))
(def _ (py/run-simple-string "with tqdm(total=100) as pbar:\n for i in range(10):\n sleep(1)\n pbar.update(10)"))
(.flush *err*)
Doing this a pressing "return" in the repl !!! afterwards
will the print the full progress in one go.
so force teh flushing of err results in a fully working progress bar, (so it prints every second for 10 seconds)
(the isatty being false or true in our code seem not make a dfference)
Yes, but it would be very useful if @cnuernber could comment why it was implemented as it is.
I can definitely say, that the change would fix the issue above, but I have no idea, if there can be any downsides.
Because then we needed to to make this controllable by the user, when he starts up libpython-clj.
The python interpreter has for example a "-u" setting for this.
I want to add as well that the behavior above is different in "embedded mode". There the flushing does happen.
It prints only after 10 second, because then by "coincidence", the JVM flushes the output stream, because the REPL prints something after every command executed. So "python by itself" cannot flush the streams out and err
Activity
jjtolton commentedon Sep 11, 2022
What is
tqdm?behrica commentedon Sep 11, 2022
https://github.com/tqdm/tqdm
Nearly every python library is using it for progress report.
behrica commentedon Sep 11, 2022
I am not sure, if something can be done on te level of
libpython-cljI think tqdm tries to detect of it has a "tty" or not.
behrica commentedon Sep 11, 2022
I think the "root cause" of the non printing of tqdm in
libpython-cljis that stderr is not a tty:while in a python shell:
behrica commentedon Sep 11, 2022
There is something strange ongoing.
the code here:
libpython-clj/src/libpython_clj2/python/io_redirect.clj
Line 37 in 11140d6
Seems to set the isatty() for both (out and err) to false.
But verifying it via "run-simple-string",
shows that err -> false, and stdin -> true
jjtolton commentedon Sep 11, 2022
Interesting. Well, that is technically correct, isn't it? Repl work with libpython-clj is not tty. Or is this for scripting?
behrica commentedon Sep 11, 2022
A python interpreter is a tty. So I would expect the same for the python process started via
libpython-cljrunning in a REPL.behrica commentedon Sep 11, 2022
I get printing by patching:
libpython-clj/src/libpython_clj2/python/io_redirect.clj
Line 34 in 11140d6
to
behrica commentedon Sep 11, 2022
This is my complete "test code", so we know what we talk about:
(run in a "fresh" repl)
behrica commentedon Sep 11, 2022
This code should print something (in realtime(, so every second for 10 seconds,
this is behavoiur in teh python interpreter.
But it does not in current `libpython-clj.
behrica commentedon Sep 11, 2022
I can "trigger" the printing manually, by "doing something" after th above lines, which make Clojure print someting to "err" or "flush it".
(it will not print every second, but all togerther at the end)
Doing this a pressing "return" in the repl !!! afterwards
will the print the full progress in one go.
-> there is flushing problem.
behrica commentedon Sep 11, 2022
Changing here:
libpython-clj/src/libpython_clj2/python/io_redirect.clj
Line 34 in 11140d6
which does "nothing" on flush into:
so force teh flushing of err results in a fully working progress bar, (so it prints every second for 10 seconds)
(the
isattybeing false or true in our code seem not make a dfference)behrica commentedon Sep 11, 2022
We have teh same issue with stdout.
This
does neither print something every second. It does so at the end (all is printed after 10 seconds)
behrica commentedon Sep 11, 2022
Doing the same fix for out (as above), makes this working:
but not
Better said: In teh python interpreter stdoud seem to be se to "auto-flush" or similar.
(but without the above patch, neither "stdout" nor "stderr" can be flushed from python code. (only from acting on the Java streams)
behrica commentedon Sep 11, 2022
So for me it seems clearly wrong to implement "flush" empty in here:
libpython-clj/src/libpython_clj2/python/io_redirect.clj
Line 34 in 11140d6
2 remaining items
behrica commentedon Sep 11, 2022
I foundsome comments on "python autoflush":
`
In fact this is the default behavior of the underlying stdio functions.
When the output is to console, the stream will be automatically flushed when a newline is encountered, but not other characters.
If the output is not a console, then even newline won't trigger a flush.
If you want to make sure about flush, you can tell the print() explicitly:
`
behrica commentedon Sep 11, 2022
The problem can be reproduced without
tqdm.The following does not print:
So even the enforcing of flushing does not make it work.
(unless I call
(.flush *err*)afterwards.)jjtolton commentedon Sep 11, 2022
Great research! Sounds like you have most of a PR ready to go for this! Would be interested to take a look and test it.
behrica commentedon Sep 12, 2022
Yes, but it would be very useful if @cnuernber could comment why it was implemented as it is.
I can definitely say, that the change would fix the issue above, but I have no idea, if there can be any downsides.
Because then we needed to to make this controllable by the user, when he starts up
libpython-clj.The python interpreter has for example a "-u" setting for this.
I want to add as well that the behavior above is different in "embedded mode". There the flushing does happen.
cnuernber commentedon Sep 13, 2022
istty and flush were stubbed out more or less to get tensorflow to work. I have no opinion on the best path here - I agree a PR would be best.
behrica commentedon Sep 13, 2022
Do you have some example code using tensorflow somewhere ?
So I can try, if it still works after change.
cnuernber commentedon Sep 13, 2022
If I remember right it was just '(import-module 'tf')`. In any case, lets get a PR and I can work with it.
behrica commentedon Sep 14, 2022
In my view , this behaves as well wrong:
It prints only after 10 second, because then by "coincidence", the JVM flushes the output stream, because the REPL prints something after every command executed. So "python by itself" cannot flush the streams out and err
cnuernber commentedon Sep 14, 2022
Ah - right - got it. I have flush added into the class now - do we need to change the istty implementation?
behrica commentedon Sep 14, 2022
good question.
In my tests I did not make a difference.
The python definition says:
Python file method isatty() returns True if the file is connected (is associated with a terminal device) to a tty(-like) device, else False.
behrica commentedon Sep 14, 2022
So either Java has something like that, which I don't think, it's the case,
Or we assume "Clojure runs mostly in REPL"... --> true
and make it configurable
behrica commentedon Sep 14, 2022
There is (System/console), not exactly the same but could be used:
https://stackoverflow.com/questions/1403772/how-can-i-check-if-a-java-programs-input-output-streams-are-connected-to-a-term
cnuernber commentedon Sep 14, 2022
OK, what I have right now is that I just implemented flush. I recommend we leave it there.
Fixes clj-python#216