Converting ANSI colored logs to documents (v. 3)

All things BZFlag - no [OT] here please
Post Reply
User avatar
alfa1
Private First Class
Private First Class
Posts: 168
Joined: Tue Dec 04, 2012 10:21 pm

Converting ANSI colored logs to documents (v. 3)

Post by alfa1 »

First, many thanks for all the views along this time (1700+ in 4,5 months)!... I hope it is being useful!
This third version is intended to be clearer and easier for reading, specially having in mind those not too advanced users, and also, because I am improving my English (thanks to BZ!) :).

--------

The BZFlag client has a command that allows us to save all the messages (all the text inside the game) into a simple text file in any moment, provided that we are logged into a server. It has 2 variations (typing previously "n" or "m" - public or team chat):

* All the commands are meaning to be without the quotation marks.

"/savemsgs" --> colored messages (using the ANSI codes)
"/savemsgs -s" --> uncolored messages ("stripping" the ANSI codes)

Then, we will obtain a file called "msglog.txt" into our BZFlag folder.

The problem with the ANSI color codes (some weird symbols and numbers inside the file) is that they ONLY run (or are shown) within terminals, for example the shell (black screen for commands) or the IRC (chat) ones; but also, if we wanted to copy/paste them into a text editor, it would be impossible. For solving this, it is necessary a program that does the conversion. First, we will use ANSIFilter for obtaining an HTML document (a common format) and, after, LibreOffice for obtaining a PDF document (an useful format for reading/annotating). Finally, we will automate the process and make some necessary changes putting all these into a shell script (program).

* We will use only Free/Libre Software (*). Please, read also the introduction of the script.

* About LibreOffice: I chose it because it was the only one option that worked and because of its quality (it has the LO team behind). All the other ones I found made the pages hard to read, by decreasing the font size when there were long lines (BZ has them often).

An example:
Example.zip
(49.17 KiB) Downloaded 368 times
Screenshots of a PDF file, including annotations:
01-02.png
(183.24 KiB) Not downloaded yet
03.png
(99.83 KiB) Not downloaded yet
04.png
(60.61 KiB) Not downloaded yet
STEPS FOR GNU/LINUX:
(For Windows, see the next post. For Mac/other systems --> TESTS ARE WELCOME! - Please, send me a P. M.)

* BZ folder: "~/.bzf/" ("~" is expanded to the home directory).

1) Download the ANSIFilter source code from http://www.andre-simon.de/doku/ansifilt ... ilter.html.

2) Extract and compile it, using the shell. Doing just a "make" command, you will obtain the executable "ansifilter" file in the "src" folder. Copy it to your BZFlag folder.

3) Copy/paste this code into a simple text editor, save it as "logconverter.sh", and make it executable (right button/Properties/Permissions/ and check "Is executable", or run "chmod +x logconverter.sh"). Please, read the important note inside the file.

Code: Select all

#!/bin/bash
#
# LOGCONVERTER 1.0 - GNU/LINUX VERSION
# ------------------------------------
#
# Shell script for converting BZFlag logs ("/savemsgs" command) from ANSI colored messages to:
#
# 1 - HTML file: you will be able to read it into any navigator and to copy/paste it to an office 
# editor like LibreOffice (a) (using ANSIFilter (b))
#
# 2 - PDF file: you will be able to read it into any PDF viewer and to add annotations if it is available, 
# as in Okular (c) (using just the LibreOffice command line converter)
# 
# *** IMPORTANT *** 
# You will need to have in the same folder of this script: the "ansifilter" program, a BZ log file 
# called "msglog.txt" and LibreOffice installed in your system BUT NOT RUNNING AT THE 
# SAME MOMENT OF THE EXECUTION
#
# Tested with BZFlag 2.4.0 (STABLE)
# Author: alpha1
#
# More info at: 
# - http://forums.bzflag.org/viewtopic.php?f=2&t=18559
# - www.bzflag.org
#
# Links:
# * All these programs, as BZFlag, are Free/Libre Software (www.fsf.org and www.gnu.org)
# (a) www.libreoffice.org
# (b) www.andre-simon.de/doku/ansifilter/en/ansifilter.html
# (c) http://okular.kde.org

# Adding top and bottom modifications:
# top) for making the title white and bold
# bottom) for setting back the default values of the console
(echo $'\e[37;1m'; cat msglog.txt; echo $'\e[0m') > temp1.txt

# Doing exchanges:
# a) black color to grey (xterm)
sed $'s/\e\[30m/\e\[38;5;248m/g' temp1.txt > temp2.txt

# b) default color to white 
sed $'s/\e\[37m\e\[0;1m/\e\[0m\e\[37;1m/g' temp2.txt > temp3.txt
sed $'s/\e\[0;1m/\e\[0m\e\[37;1m/g' temp3.txt > temp4.txt

# Creating the HTML file and setting the document title
./ansifilter -d msglog --html temp4.txt > temp.html

# Changing the background color to grey for the annotations can be seen
sed $'s/<body>/<body bgcolor="#666666">/' temp.html > msglog.html

# Deleting the temp files
rm temp*

# Generating the PDF file (*** optional but recommended - it can be commented ***)
libreoffice --headless -convert-to pdf msglog.html

# Deleting the "msglog.html" file (*** optional - it can be uncommented ***)
# rm msglog.html
4) You can run now the script by doing:

"./logconverter.sh" (without any arguments)


Enjoy!

alpha


:!: Attention: don't use it with bad purposes! Remember that the servers save logs too and you can't lie against them.

:idea: (*) Free/Libre Software: You have the rights to use, modify and redistribute it freely, with some restrictions. More info at:
http://www.fsf.org
http://www.gnu.org

* Note: the message window in BZFlag has a limited memory size (called "cache memory"). As the "/savemsgs" command will ALWAYS use the "All" mode ("all kind of messages"), the amount of pages you are seeing in your screen is what you will get saved. You can use it several times, though, and the new text will be always added at the bottom part. If you lost some chat messages, you can switch to the "only Chat mode" (Shift+F2) and use, instead, the screenshots (F5).
Last edited by alfa1 on Thu Feb 06, 2014 8:16 am, edited 99 times in total.
alezakos
Private First Class
Private First Class
Posts: 82
Joined: Thu Sep 16, 2010 1:24 pm

Re: Converting ANSI colored logs to documents

Post by alezakos »

This is great, and it could definitely make reading the chat logs more fun!
brad
Private First Class
Private First Class
Posts: 689
Joined: Sun May 08, 2005 12:26 pm
Location: UK
Contact:

Re: Converting ANSI colored logs to documents - [RE-WRITTEN

Post by brad »

Nice work alpha
User avatar
alfa1
Private First Class
Private First Class
Posts: 168
Joined: Tue Dec 04, 2012 10:21 pm

Re: Converting ANSI colored logs to documents (v. 3)

Post by alfa1 »

Thanks for the encouragements by here and by the PMs, and for the received help! I hope this method is useful, both for the personal and for the admin/cop uses. Greetings!

---------

STEPS FOR WINDOWS:

* BZ folder: The "msglog.txt" file is saved 1 level up from the config file folder (http://wiki.bzflag.org/Config_File).

* VERSION 1.0-BETA-1 - FEBRUARY 03, 2014.
*** TESTS ARE WELCOME *** --> Please, send me a P.M.

* Wine test (v. 1.7.7 - Windows 9x/NT/2000/XP, Windows 3.x y DOS): success.

1) Requirements:
2) Copy/paste this code into a simple text editor and save it as "logconverter.bat" into the log folder. Please, read the important note inside the file.

Code: Select all

@echo off
::
:: LOGCONVERTER 1.0-BETA-1 - WINDOWS VERSION
:: -----------------------------------------
::
:: Batch script for converting BZFlag logs ("/savemsgs" command) from ANSI colored messages to:
::
:: 1 - HTML file: you will be able to read it into any navigator and to copy/paste it to an office 
:: editor like LibreOffice (a) (using ANSIFilter (b))
::
:: 2 - PDF file: you will be able to read it into any PDF viewer and to add annotations if it is available, 
:: as in Okular (c) (using just the LibreOffice command line converter)
::
:: *** IMPORTANT ***:
:: a - You will need to have in the same folder of this script the "ansifilter.exe" program, the "mingwm10.dll"
:: and "libgcc_s_dw2-1.dll" libraries and a BZ log file called "msglog.txt"
:: b -  You will need to have installed in your system both the sed tool and, optionally but recommended,
:: the LibreOffice suite - THE LAST ONE MUST NOT BE RUNNING AT THE SAME MOMENT OF THE EXECUTION
::
:: Tested with BZFlag 2.4.0 (STABLE)
:: Author: alpha1
::
:: *** TEST ARE WELCOME ***
:: Wine test (Windows 9x/NT/2000/XP, Windows 3.x y DOS): success
::
:: More info at:
:: - http://forums.bzflag.org/viewtopic.php?f=2&t=18559
:: - http://www.bzflag.org
::
:: Links:
:: * All these programs, as BZFlag, are Free/Libre Software (http://www.fsf.org/www.gnu.org)
:: (a) http://www.libreoffice.org
:: (b) http://www.andre-simon.de/doku/ansifilter/en/ansifilter.html
:: (c) http://okular.kde.org

:: Setting locally the programs paths for sed and LibreOffice in the PATH environment variable
:: It is temporal, just until the script ends
:: *** Edit in case of not working ***
setlocal
set PATH=%PATH%;C:\Program Files (x86)\GnuWin32\bin\;C:\Program Files (x86)\LibreOffice 4\program\

:: Adding top and bottom modifications
:: top) for making the title white and bold
:: bottom) for setting back the default values of the console
echo [37;1m > temp1.txt
type msglog.txt >> temp1.txt
echo [0m >> temp1.txt

:: Doing exchanges:
:: a) black color to grey (xterm)
sed.exe "s/\[30m/\[38;5;248m/g" temp1.txt > temp2.txt

:: b) default black to white
sed.exe "s/\[37m\[0;1m/\[0m\[37;1m/g" temp2.txt > temp3.txt
sed.exe "s/\[0;1m/\[0m\[37;1m/g" temp3.txt > temp4.txt

:: Creating the HTML file and setting the document title
ansifilter.exe -d msglog --html temp4.txt > temp.html

:: Changing the background color to grey for the annotations can be seen
sed.exe "s/<body>/<body bgcolor=#666666>/" temp.html > msglog.html

:: Deleting temp files
del temp*.*

:: Generating the PDF file (*** optional but recommended - it can be commented ***)
soffice.exe --headless --convert-to pdf msglog.html

:: Deleting the "msglog.html" file (*** optional - it can be uncommented ***)
:: del msglog.html
3) You can run now the script just by doing double click on it or by opening a terminal (Start Menu/Run/cmd.exe), going to the log folder (using the "cd" command) and typying "logconverter". If you use the console way, you will be able to see the errors output.

--------

Other useful GNU commands/tools for Windows, including Coreutils:
http://gnuwin32.sourceforge.net
Last edited by alfa1 on Mon Mar 17, 2014 9:24 pm, edited 59 times in total.
brad
Private First Class
Private First Class
Posts: 689
Joined: Sun May 08, 2005 12:26 pm
Location: UK
Contact:

Re: Converting ANSI colored logs to documents - [RE-WRITTEN

Post by brad »

This might make things a bit easier for those too lazy to install the above:

http://bz.ninja.so/

Example at http://bz.ninja.so/u/242f35d998cef8b546 ... fdbb5.html
User avatar
alfa1
Private First Class
Private First Class
Posts: 168
Joined: Tue Dec 04, 2012 10:21 pm

Re: Converting ANSI colored logs to documents (v. 3)

Post by alfa1 »

After some important changes made in the BZ code related to "/savemsgs", I am updating the script.

Changes made:
- After the addition of timestamps in log names --> I group all logs into only one file called "msglog.txt" (original files keep unmodified). (Also the "-t" option was added to "/savemsgs" which saves timestamps to logs.)
- After switching the majority of the colors from ANSI to ANSI-RGB systems --> As a first approach to solve this, I convert back, from ANSI-RGB to ANSI (or xterm) systems, the main colors (used by players): yellow #1 (rogue), red, green, blue, purple, white #1 (observer), grey #1 (rabbit) and orange (hunter) (cyan, yellow #2 and white #2 (gral. messages), and black (converted, later, to xterm grey #2 by me) keep being ANSI). IMPORTANT: If any of these main colors is customized (to a *different* color from an ANSI one), it will be converted to white.
(Numers #1 and #2 do not necessarily mean different colors, but treaten differently by the BZ code. In example, you can change the yellow of rogue players, but can't change the yellow of gral. messages.)

The main colors (used by players) can be changed by using "/localset <variable> [value]" (without externals quotation marks nor brackets; value needs quotation marks) like this:
- variable:
team tank colors: roguecolor, redcolor, greencolor, bluecolor, purplecolor, observercolor, rabbitcolor, huntercolor.
team radar colors: rogueradar, redradar, greenradar, blueradar, purpleradar, observerradar, rabbitradar, hunterradar.
- value: the arithmetic notation is used to write the RGB color; the values can be decimal and go from 0 to 1. Type "/localset <variable>" to see its current values. More info at https://en.wikipedia.org/wiki/RGB_color ... sentations .

In example:
to make red (1, 0, 0) players become yellow (1, 1, 0) in HUD:
/locaset redcolor "1 1 0"
to make green (0, 1, 0) players become sepia (0.44, 0.26, 0.08) in radar:
/locaset greenradar "0.44 0.26 0.08"

Linux code:

Code: Select all

#!/bin/bash
#
# LOGCONVERTER 2.0-BETA-1 - GNU/LINUX VERSION
# -------------------------------------------
#
# Shell script for converting BZFlag logs ("/savemsgs" command) from ANSI colored messages to:
#
# 1 - HTML file: you will be able to read it into any navigator and to copy/paste it to an office
# editor like LibreOffice (a) (using ANSIFilter (b)).
#
# 2 - PDF file: you will be able to read it into any PDF viewer and to add annotations if it is available,
# as in Okular (c) (using just the LibreOffice command line converter).
#
# *** IMPORTANT ***
# You will need to have in the same folder of this script: the "ansifilter" program, one or more log files
# (either an old-style-filename one ("msglog.txt") or one or more new-style-filename ones
# ("msglog-<date>_<time>.txt")) and LibreOffice installed in your system BUT NOT RUNNING AT THE SAME
# MOMENT OF THE EXECUTION.
#
# NOTE ABOUT THIS VERSION - FIRST APPROACH TO THE RGB-ANSI ISSUE:
# --------------------------------------------------------------
# After some important changes made in the BZ code, I am updating the script. 
#
# CHANGES MADE:
# ------------
# - After the addition of TIMESTAMPS in log names --> I group all logs into only one file called "msglog.txt" 
# (original files keep unmodified). (Also the "-t" option was added to "/savemsgs" which saves timestamps
# to logs.)
# - After switching the majority of the colors FROM ANSI TO ANSI-RGB SYSTEMS --> As a first approach
# to solve this, I convert back, from ANSI-RGB to ANSI (or xterm) systems, the main colors (used by 
# players): yellow #1 (rogue), red, green, blue, purple, white #1 (observer), grey #1 (rabbit) and 
# orange (hunter) (cyan, yellow #2 and white #2 (gral. messages), and black (converted, later, to 
# xterm grey #2 by me) keep being ANSI). IMPORTANT: IF ANY OF THESE MAIN COLORS IS CUSTOMIZED (TO A
# *DIFFERENT* COLOR FROM AN ANSI ONE), IT WILL BE CONVERTED TO WHITE.
#
# Tested with BZFlag 2.4.6 (MAINT)
# Author: alpha1
#
# More info at:
# - http://forums.bzflag.org/viewtopic.php?f=2&t=18559
# - www.bzflag.org
#
# Links:
# * All these programs, as BZFlag, are Free/Libre Software (www.fsf.org and www.gnu.org):
# (a) www.libreoffice.org
# (b) www.andre-simon.de/doku/ansifilter/en/ansifilter.html
# (c) http://okular.kde.org


# *** Start of addition for version 2.0-BETA-1 ***

# Checking either if a "msglog.txt" doesn't exist together with the time stamped ones and if there is not
# any log file

# Flag time stampèd file
flagts=0
# Flag old file
flago=0

for i in msglog-*.txt
do
  if [ $i != "msglog-*.txt" ]
  then
    flagts=1
  fi
done

if [ -f "msglog.txt" ]
then
  flago=1
fi

if [ $flagts -eq 1 ]
then
  if [ $flago -eq 1 ]
  then
    echo "ERROR: a \"msglog.txt\" file already exists together with the time stamped ones."
    exit
  fi
elif [ $flago -eq 0 ]
  then
    echo "ERROR: neither a \"msglog.txt\" file nor the time stamped ones exist."
    exit
fi

# Concatenating msglogs - only for the new timestamps ones
if [ $flagts -eq 1 ]
then
  for i in msglog-*.txt
  do
    cat $i >> msglog.txt
  done
fi

# Making temporal replacement of main colors from ANSI-RGB color codes to only ANSI/xterm ones:
# (Color names taken from https://en.wikipedia.org/wiki/Lists_of_colors)
		
# 1 - yellow #1 (rogue)
sed $'s/\e\[38;2;255;255;0m/\e\[33m/g' msglog.txt > temp0-1.txt

# 2 - red
sed $'s/\e\[38;2;255;0;0m/\e\[31m/g' temp0-1.txt > temp0-2.txt

# 3 - green
sed $'s/\e\[38;2;0;255;0m/\e\[32m/g' temp0-2.txt > temp0-3.txt

# 4 - blue (no name)
sed $'s/\e\[38;2;25;51;255m/\e\[34m/g' temp0-3.txt > temp0-4.txt

# 5 - fuchsia (purple)
sed $'s/\e\[38;2;255;0;255m/\e\[35m/g' temp0-4.txt > temp0-5.txt

# 6 - white #1 (observer)
sed $'s/\e\[38;2;255;255;255m/\e\[37m/g' temp0-5.txt > temp0-6.txt

# 7 - grey #1 (rabbit) (no name; almost X11 gray)
sed $'s/\e\[38;2;204;204;204m/\e\[38;5;252m/g' temp0-6.txt > temp0-7.txt

# 8 - orange (hunter)
sed $'s/\e\[38;2;255;127;0m/\e\[38;5;208m/g' temp0-7.txt > temp0-8.txt

# *** End of addition for version 2.0-BETA-1 (except for the "temp0-8.txt" filename behind) ***


# Adding top and bottom modifications:
# top) for making the title white and bold
# bottom) for setting back the default values of the console
(echo $'\e[37;1m'; cat temp0-8.txt; echo $'\e[0m') > temp1.txt

# Doing exchanges:
# a) black color to grey (xterm)
sed $'s/\e\[30m/\e\[38;5;248m/g' temp1.txt > temp2.txt

# b) default color to white
sed $'s/\e\[37m\e\[0;1m/\e\[0m\e\[37;1m/g' temp2.txt > temp3.txt
sed $'s/\e\[0;1m/\e\[0m\e\[37;1m/g' temp3.txt > temp4.txt

# Creating the HTML file and setting the document title
./ansifilter -d msglog --html temp4.txt > temp.html

# Changing the background color to grey for the annotations can be seen
sed $'s/<body>/<body bgcolor="#666666">/' temp.html > msglog.html

# Deleting the temp files
rm temp*

# Generating the PDF file (*** optional but recommended - it can be commented ***)
libreoffice --headless -convert-to pdf msglog.html

# Deleting the "msglog.html" file (*** optional - it can be uncommented ***)
# rm msglog.html
Note #1: switching from ANSI to ANSI-RGB systems for consoles/terminals, as far as I could research, is not completely accepted worldwide (some accept it, some others don't). In the other hand, the RGB system has already lots of years in the market. What I can see here is that we have 2 "worlds" represented: an older "world" of computer terminals/consoles (using an older ANSI system) and a newer "world" of (some) emulators of those consoles/terminals (using a newer RGB system). Into the BZ code, regarding to the console emulator, developers made a switch from ANSI to RGB which seems to be a mixing between both systems. Since ANSIFilter (the heart of my script), despite of being updated periodically, has not included RGB into it still (not sure if they will do it in the future), I decided to update the script with this first approach, to make it still operative. The next step should be one of these: either finding another tool/new compatible version of ANSIFilter (supporting both ANSI and RGB systems) or developing (by me or other person) a definitive solution.

Note #2: In case the version you use is not totally compatible with the 2.4.6 version (about colors), check [BZFlag source folder]/src/bzflag/defaultBZDB.cxx, around the line 150 ("team tank colors" and "team radar colors" in this version), see what colors you use instead and replace them into the logconverter code.
Last edited by alfa1 on Sat Aug 20, 2016 3:19 am, edited 2 times in total.
User avatar
blast
General
General
Posts: 4931
Joined: Fri Mar 21, 2003 3:49 pm
Location: playing.cxx
Contact:

Re: Converting ANSI colored logs to documents (v. 3)

Post by blast »

Please note that you'll need to specify what license you are releasing these under. As of yet you've not actually mentioned any specific license. You might want to read through this:
https://www.gnu.org/licenses/license-re ... tions.html

And more general information about licensing:
http://www.fsf.org/licensing/
"In addition to knowing the secrets of the Universe, I can assure you that I am also quite potty trained." -Koenma (Yu Yu Hakusho)

Image
Post Reply