Misc Backtraces

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 12:54 am

I'm creating a backtrace thread so those who care can have something to look at... Deal with it. Administrators are encouraged to edit and move this as desired.

I freakin' DARE anyone and everyone that runs a server to join me in posting backtraces gathered from core dumps when running a server and it crashes. Server owners who wish to contribute back by submitting such information should follow the following information accordingly.

Linux users...

DTRemenak wrote:
I_Died_Once wrote:I keep getting segmentation faults. I'm not sure if its the plugins or the CVS version. I will say that I am running all plugins on Planet MoFo for the time being.
When a CVS version crashes, backtraces are very helpful.

First look for a corefile. Usually they're dumped in whatever directory you ran the bzfs command from (probably your home directory). Should be named "bzfs.core". If you can't find a corefile and you're running linux, run "ulimit -a".[Note 1] If "core file size" is 0 then run "ulimit -c 100000", which will tell it it can use up to 100mb (assuming 1k blocks) for corefiles, then restart bzfs. Wait for it to crash again and when it does:

Assuming you have gdb installed, and bzfs crashes and dumps a corefile, do "gdb --core=bzfs.core". When you get the "gdb>" prompt enter "file /path/to/bzfs", wait for it to load the symbols, then enter "bt". It should spew about ten or a dozen lines starting with "#0" through "#n", that's the backtrace.[Note 2] If you can post the backtrace or send it to any of the devs we can try to nail down your crash. Don't send the corefile itself to anyone...since you build your own binaries only your computer can make any sense of them.

BTW: same instructions go for anyone else experiencing crashes on unix-like platforms. I should probably post it somewhere more visible.

Note 1: Other systems and some strange linux distros will use "limit" instead of "ulimit", and the syntax of "limit" may vary (use the manpage)
Note 2: Use "quit" at the "gdb>" prompt to exit gdb.

Someone should post these instructions for windows and mac users so no one can say the information is not out there and organized.

If everyone posted the information every time their server crashed, perhaps we could speed up and give back to development of the game altogether. Woot.

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 12:59 am

Now, for my first backtrace...

this morning's CVS, the listed plugins below, free for all...

killall.so
playHistoryTracker.so
chathistory.so
fixedspawn.so
scorereset.so
airspawn.so,25
RogueGenocide.so
vocaliser.so,/home/death/bzflag/plugins/vocaliser/voices.txt
Janitor.so,/home/death/bzflag/plugins/Janitor/config
bzfscron.so,/home/death/bzflag/plugins/bzfcron/crontab

I was testing out the rogue genocide plugin, which, WORKS! Its kills rogues as a team when hit with genocide! BOth when hit from a regular team tank, AND when hit by a fellow rogue! It says "you were teh victim of rogue genocide" and it rocks.

I was a rogue, and I wanted to see what happened when I hit myself with my own genocide. I killed myself with geno, and the server crashed. Here are the results:

Code: Select all

Player Triger Happy (HEHEHEHE) [0] uread() 168.103.225.239:49268 len 31 from 168.103.225.239:49268 on 4
Player hannahkiller [6] uread() 59.136.56.137:62607 len 31 from 59.136.56.137:62607 on 4
Player The Spanish Inquisition! [4] uread() 67.42.47.47:50227 len 31 from 67.42.47.47:50227 on 4
Player wren [3] uread() 84.9.88.91:42782 len 33 from 84.9.88.91:42782 on 4
bzfscron: tick!
Player Triger Happy (HEHEHEHE) [0] uread() 168.103.225.239:49268 len 31 from 168.103.225.239:49268 on 4
Player wren [3] uread() 84.9.88.91:42782 len 33 from 84.9.88.91:42782 on 4
Player Darth Vader [2] uread() 192.168.1.1:53234 len 9 from 192.168.1.1:53234 on 4
Segmentation fault (core dumped)
death@linux:~> bzflag -solo 5
death@linux:~> gdb
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403f2116 in ?? ()
(gdb) file /home/admin/bzflag/bin/bzfs
/home/admin/bzflag/bin/bzfs: No such file or directory.
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403f2116 in ?? ()
#1  0x08174e3b in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081cd428 in ?? ()
#3  0xbfffdf70 in ?? ()
#4  0x00000002 in ?? ()
#5  0xbfffdf70 in ?? ()
#6  0xbfffdeb8 in ?? ()
#7  0x080d373d in WorldEventManager::callEvents (this=0x81c7300, eventType=136098048, eventData=0x7) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb) quit
death@linux:~>

Post by **DTRemenak** » Sat Aug 27, 2005 2:03 am

(This post contains instructions in general, not specific to I_Died_Once, who has been doing a great job. Feel free to just do the "monkey-see monkey-do" with what he does instead of reading all this nonsense.)

As an addendum, if you get a backtrace with this line in it:

I_Died_Once wrote:Previous frame inner to this frame (corrupt stack?)

they aren't generally very useful. All it tells us is that we have some kind of problem with a stack-smashing crash, the rest of the information is most likely corrupted. I_Died_Once provided enough information that we know it's probably caused by a Rogue with RogueGenocide suiciding, so we can test that case. If you know how to run valgrind (http://valgrind.org/) (or purify if you have it) against your server, that will tell us exactly what the problem is (feel free to post valgrind logs), but it is not a "simple" thing to do and will absolutely slaughter your performance (think 1000% to 3000% more cpu usage, and those are not typos). If you don't think we know that there's a stack-smashing crash out there, post that you got one and your configuration (full version, loaded plugins, etc). If in doubt, post. The more information, the better. It's better to get duplicates than not be aware of a crash condition.

Another useful bit of information is the last couple lines of debug output from the server, if you're running with -d options. Feel free to trim sensitive information (e.g. /identify), but leave the command (so "Player [1] DTRemenak -> All: /identify <snip>" is all we need) since that might be relevant. If you know what the problem is a short description is useful. Also please include the source date (e.g. "Today's CVS" or "2.0.3.20050813" is good), since that's the only way we know what the line numbers in the backtrace actually mean

.

Instructions for obtaining a backtrace on the mac should be identical to those on linux. Macs with the dev system installed have gdb also, and dump core the same way AFAIK. On windows you basically can't get a backtrace, 'cause there's no core dump facility, and we don't even build bz on windows with debug information in it. Upgrade to linux. Err, wrong topic, sorry...

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 2:12 am

I had to re-write a part of my conf file and I forgot to exclude scorereset. I dont know if that was it, but I'll post the next backtracae that occurs.

Another good idea is to delete the core dumps once you get the backtrace out of it.
That way there is no question as to if it is being over written or not.

A Meteorite · Post by **A Meteorite** » Sat Aug 27, 2005 3:04 am

I'll start backtracing my server too. Once I get it on Linux, though... is it possible to do this on a Mac? (I know you can't with plugins, but what about back traces?)

Tropican8 · Post by **Tropican8** » Sat Aug 27, 2005 3:07 am

DTRemenak wrote:we don't even build bz on windows with debug information in it.

What about people who use MinGW? It builds BZFlag with debug info. Can someone who builds with MinGW from CVS somehow do this also?

Post by **DTRemenak** » Sat Aug 27, 2005 3:29 am

A-Meteorite: gdb works just the same on a mac as on linux, assuming you're building from source and not using icky app bundle gadgets. If you use limit or ulimit (don't know which the mac uses) as in my original post, you can get corefiles, load them up, print out the backtrace just like any other unixlike system.

Tropican: There's no core dump facility in mingw. You could get a backtrace without needing a corefile if you ran bzfs directly in gdb ("gdb bzfs -conf my.conf"), then when it crashes you can just type "bt" at the gdb prompt. Disadvantages to that are 1) gdb is running the whole time, eating up your machine's resources, and 2) you can't redirect bzfs' output to a log file. An advantage is that it is guaranteed to load the plugin symbols, so you can potentially nail down a crash that is deeper in a plugin with that method (that goes for all os'es).

BTW: I've done an experiment and it does seem that gdb will let you load plugin debug info in addition to bzfs'. So if you have a crash and get a lot of "??? (someplugin.so)" entries when you ask for a backtrace, you can do "add-symbol-file someplugin.so" after "file bzfs" and it will grab all the symbols. That can be very helpful if a plugin crashes and you have an intact stack (if you have a smashed stack it still won't help).

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 7:43 am

Code: Select all

Player Capn Kitty [4] uread() 69.17.103.104:4523 len 31 from 69.17.103.104:4523 on 4
Player Capn Kitty [4] uread() 69.17.103.104:4523 len 47 from 69.17.103.104:4523 on 4
Player Darth Vader [1] uread() 192.168.1.1:59074 len 31 from 192.168.1.1:59074 on 4
Player Capn Kitty [4] uread() 69.17.103.104:4523 len 47 from 69.17.103.104:4523 on 4
Player Capn Kitty [4] uread() 69.17.103.104:4523 len 47 from 69.17.103.104:4523 on 4
Player jlw [7] uread() 209.157.141.69:53008 len 31 from 209.157.141.69:53008 on 4
Player Darth Vader [1] uread() 192.168.1.1:59074 len 31 from 192.168.1.1:59074 on 4
Player Darth Vader [1] uread() 192.168.1.1:59074 len 31 from 192.168.1.1:59074 on 4
Player Darth Vader [1] uread() 192.168.1.1:59074 len 9 from 192.168.1.1:59074 on 4
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403f2116 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403f2116 in ?? ()
#1  0x08174e3b in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081d52f0 in ?? ()
#3  0xbfffe3d0 in ?? ()
#4  0x00000002 in ?? ()
#5  0xbfffe3d0 in ?? ()
#6  0xbfffe318 in ?? ()
#7  0x080d373d in WorldEventManager::callEvents (this=0x81d6160, eventType=136142768, eventData=0x6) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

Tropican8 · Post by **Tropican8** » Sat Aug 27, 2005 4:26 pm

DTRemenak wrote:There's no core dump facility in mingw.

Does this work?

http://members.telering.at/jessich/ming ... wdump.html

Post by **DTRemenak** » Sat Aug 27, 2005 5:24 pm

That sounds like it might work. Not entirely sure but it should.

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 5:37 pm

This one happened when I wasn't around. No telling what the fools on were up to. I know its nothing that'll provide any help, but hey - here it is anyway

Code: Select all

Player BPDSMS [3] uread() 66.9.253.2:1331 len 31 from 66.9.253.2:1331 on 4
Player mps [4] uread() 80.98.242.128:1063 len 31 from 80.98.242.128:1063 on 4
Player BPDSMS [3] uread() 66.9.253.2:1331 len 31 from 66.9.253.2:1331 on 4
Player BPDSMS [3] uread() 66.9.253.2:1331 len 31 from 66.9.253.2:1331 on 4
Player mps [4] uread() 80.98.242.128:1063 len 31 from 80.98.242.128:1063 on 4
Player BPDSMS [3] uread() 66.9.253.2:1331 len 31 from 66.9.253.2:1331 on 4
Player Avatar [5] uread() 71.112.162.205:4730 len 31 from 71.112.162.205:4730 on 4
Player 0013 [6] uread() 70.179.86.145:60462 len 31 from 70.179.86.145:60462 on 4
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403f2116 in ?? ()
(gdb) file /home/death/bzflag/bn/bzfs
/home/death/bzflag/bn/bzfs: No such file or directory.
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403f2116 in ?? ()
#1  0x08174e3b in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081e53a8 in ?? ()
#3  0xbfffe3d0 in ?? ()
#4  0x00000002 in ?? ()
#5  0xbfffe3d0 in ?? ()
#6  0xbfffe318 in ?? ()
#7  0x080d373d in WorldEventManager::callEvents (this=0x81e5270, eventType=136190952, eventData=0x2) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 6:40 pm

It gave me this odd error just now over playHistoryTracker

Code: Select all

./bzfs: symbol lookup error: /home/death/bzflag/plugins/playHistoryTracker/playHistoryTracker.so: undefined symbol: _ZN11bzApiString5c_strEv

Thats something like bzfscron gave me before it was fixed...

It ran fine until I joined, thats when it crapped out and gave me the error. I will try re-making the plugin, even though the API hasnt been updated....

...and re-making it seemed to fix it off the bat. I

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 6:40 pm

Code: Select all

Player Darth Vader [0] uread() 192.168.1.1:52475 len 31 from 192.168.1.1:52475 on 4
Player Darth Vader [0] uread() 192.168.1.1:52475 len 31 from 192.168.1.1:52475 on 4
Player Darth Vader [0] uread() 192.168.1.1:52475 len 9 from 192.168.1.1:52475 on 4
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403ee116 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403ee116 in ?? ()
#1  0x08174ebb in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081c93b8 in ?? ()
#3  0xbfffe3d0 in ?? ()
#4  0x00000002 in ?? ()
#5  0xbfffe3d0 in ?? ()
#6  0xbfffe318 in ?? ()
#7  0x080d36ed in WorldEventManager::callEvents (this=0x81b8470, eventType=1077929732, eventData=0x2) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

Post by **DTRemenak** » Sat Aug 27, 2005 8:02 pm

The API was actually changed this morning. It will not affect sources but you will have to recompile any plugins that use bzApiString.c_str(), which is now const-safe.

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 10:48 pm

I could be wrong, I more than likely am - but this happened right after bzfscron did a message.

The only plugins loaded were airspawn, bzfscron, and RogueGenocide. Genocide hadn't recently been fired, and has been runing significantly more smooth. I got bzfscron in my own test mode, doing a message every minute, every other minute, every three minutes, and every four minutes. I tried one for thirty seconds, but I think it translated into every five. (I did a */.5 in the crontab)

OK, the dump, which according to an earlier post is unuseable. Delete it if its no good to ya. (Or just purge this once in a while, I'm gonna flood you peeps)

Code: Select all

Player Hunter [7] uread() 217.252.117.162:1067 len 31 from 217.252.117.162:1067 on 4
Player Holey Tank [4] uread() 69.228.220.238:62106 len 34 from 69.228.220.238:62106 on 4
Player Darth Vader [2] uread() 192.168.1.1:48974 len 31 from 192.168.1.1:48974 on 4
Player undertaker [8] uread() 213.229.7.134:33649 len 31 from 213.229.7.134:33649 on 4
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403db116 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403db116 in ?? ()
#1  0x08174ebb in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081ba400 in ?? ()
#3  0xbfffe1a0 in ?? ()
#4  0x00000001 in ?? ()
#5  0xbfffe1a0 in ?? ()
#6  0xbfffe0e8 in ?? ()
#7  0x080d36ed in WorldEventManager::callEvents (this=0x81a6c2c, eventType=bz_eNullEvent, eventData=0x6) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

RPG · Post by **RPG** » Sat Aug 27, 2005 11:10 pm

I_Died_Once, one of the plugins you're running is toppling your stack. Every report has this in it:

Code: Select all

Previous frame inner to this frame (corrupt stack?)

I tell you, one of them is dirty. A spy. A leak.

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 11:35 pm

Another corrupt stack - I only had ROgue Geno and airspawn loaded.

Code: Select all

Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403db116 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403db116 in ?? ()
#1  0x08174ebb in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081b8420 in ?? ()
#3  0xbfffdf70 in ?? ()
#4  0x00000001 in ?? ()
#5  0xbfffdf70 in ?? ()
#6  0xbfffdeb8 in ?? ()
#7  0x080d36ed in WorldEventManager::callEvents (this=0x55e2ab45, eventType=136041632, eventData=0x2) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

I_Died_Once · Post by **I_Died_Once** » Sat Aug 27, 2005 11:40 pm

Fine, RPG - next go 'round will be ONLY the rogue geno

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 1:02 am

OK - The only plugin loaded was RogueGenocide. Re-did the flag scheme for this, as well as modified team limits (two red, lots of rogue)

And I learned something. If it isn't documented, then let it be known... In order to have a genocide flag on the field at all - plugin or not - there must be at least two members of a real team allowed in the conf file. I thought I broke my file and had to refer myself to an online backup of the conf.

It still says the stack is corrupt.

I am going to run all the others back on and turn this one off after this

Here, only rogue geno loaded:

Code: Select all

Player  [0] outbound UDP up
Player come and get me... I dare U [0] sent version string: 2.0.0.20050117-STABLE-W32VC71
Player come and get me... I dare U [0] has joined from 4.247.155.207 at 2005-08-27 20:56:09 with token ""
Player come and get me... I dare U [0] removed at 2005-08-27 20:56:09: /rejected
Player gangbang [1] removed at 2005-08-27 20:56:16: left
Queuing ADD message to list server
Player [0] accept() from 69.28.129.166:40986 on 6
Player [0] submitted reverse resolve query
Player  [0] removed at 2005-08-27 20:56:16: Disconnected
Player [0] accept() from 4.247.155.207:2361 on 6
Player [0] submitted reverse resolve query
Player [0] resolved to dialup-4.247.155.207.Dial1.Tampa1.Level3.net
Player  [0] inbound UDP up 4.247.155.207:2361 actual 2361
Player  [0] outbound UDP up
Player come and get me... I dare U [0] sent version string: 2.0.0.20050117-STABLE-W32VC71
Player come and get me... I dare U [0] has joined from 4.247.155.207 at 2005-08-27 20:56:18 with token ""
Queuing ADD message to list server
Player [1] accept() from 69.28.129.166:40989 on 7
Player [1] submitted reverse resolve query
Player  [1] removed at 2005-08-27 20:56:19: Disconnected
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `./bzfs -conf FFA.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x403d8116 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0x403d8116 in ?? ()
#1  0x08174ebb in __gthread_active_p()::__gthread_active_ptr ()
#2  0x081bb280 in ?? ()
#3  0xbfffe3d0 in ?? ()
#4  0x00000001 in ?? ()
#5  0xbfffe3d0 in ?? ()
#6  0xbfffe318 in ?? ()
#7  0x080d36ed in WorldEventManager::callEvents (this=0x81aa358, eventType=bz_eNullEvent, eventData=0x1) at stl_vector.h:501
Previous frame inner to this frame (corrupt stack?)
(gdb)

Post by **DTRemenak** » Sun Aug 28, 2005 3:03 am

Gah. I don't understand that. I just finished running about two hours of rogue geno tests, to no avail - I can't get it to crash at all, or even trigger any valgrind errors.

If you would like to run your server under valgrind in a configuration that you know crashes, and you have (a little) time and (a lot of) cpu power and memory to spare, here's a quickie setup:
* If you don't want to do this you don't need to, just skip the rest of the post. We will probably end up finding out why RogueGenocide crashes either way, but this may make it sooner rather than later.
* Make sure I'm not wrong, and you are actually running linux kernel 2.4 or 2.6 (basically any linux from the last couple years), on a 32-bit, intel or amd-processor box (not amd64 or em64t, and certainly not a mac). "uname -srm" will give you the info if you don't know it (something like "Linux 2.4.29 i686" is fine)
* Make sure you have enough cpu power and memory to spare. If you don't, bzfs will end up lagging out on your players, which is uncool. These are guidelines, and you can feel free to push the boundaries, it's not like it'll all of a sudden wreck the server if you're a couple % off. You should have about 10 times as much cpu power as bzfs typically uses, and about 5 times as much memory as bzfs typically uses, free. So if bzfs uses 8% cpu and 13 mb of memory you should be using a total of no more than 20% of your box' cpu and should have at least 65mb of free memory before you start. You can check all that with the "top" command if you don't have some other tool you already know or prefer.
* Install valgrind 2.4 from http://valgrind.org/. These instructions might not work with valgrind 3, I haven't downloaded that yet myself

. If you can't get it compiled or installed, feel free to ask for help.
* Run "valgrind --gen-suppressions=no --db-attach=no --leak-check=no --logfile=bzfslog bzfs -conf /path/to/your/configuration.conf" instead of "bzfs -conf /path/to/your/configuration.conf"
* When bzfs crashes, you'll find a file called "bzfslog.pidXXXXX" (where the X's are numbers) in the current directory. That's the valgrind log. Attach it to a post here (it's likely going to be too big to paste in), or email it to someone.
* After you have a log with information in it posted, you can go back to running bzfs normally.

That won't be quite as useful as a proper valgrind setup but it should give enough information. If you'd like to make it more useful, recompile bzflag after doing "./configure --enable-debug" (that's bzflag's ./configure not valgrind's). Ideally you'd also be able to create a proper suppressions list and we would get just the error that actually causes the crash (bzfs should be valgrind-clean), along with a valgrindified backtrace, but that's a long process that involves a decent amount of educated guessing. Actually, since you're running SuSE (iirc) valgrind may ship with a matching suppression list (it'll load it automatically), which would be cool.

If you don't want to take the time or effort to valgrind it (or if the machine your server is running on isn't up to it) that's cool too, but RogueGenocide will probably keep crashing for you until we can get it to crash for us. This is strictly optional, and it's admittedly harder than just pulling a backtrace

.

This post is too long. I've edited it to simplify some things.

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 8:02 am

OK, I did download the valgrind 3.0.0 source...

...but one of my YAST sources for Suse (install stuff soooooo easy if it is in the open source category yusually) and they have valgrind 2.4I am installing 2.4 with ease and will use it...

Give me time to get it configured and for me to read up and learn this new program.

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 3:47 pm

FYI - Before I left the house this morning (I've been dead tired and not up to posting anything with substance) I did the following...

Changed the server to have only geno flags (makes it easier)
Only the Rogue Genocide is loaded
Yesterday's CVS
Two red tanks and seven (or nine?) rogues allowed. Plus some observers

I installed valgrine via YAST, version 2.4 as I mentioned above. (I got 3.0's source downloaded)

...and I have valgrind actualy right now. I changed path's, and with the way I got stuff linked (valgrind's docs says it should recognize that with no issue) my command to start the server was

Code: Select all

valgrind --gen-suppressions=no --db-attach=no --leak-check=no -- ./bzfs -conf /FFA.conf

Now I must tell ya - it did NOT liek the logifle option you had in there. I went so far as to create a text file for this purpose, and I gave it the full path as well. Tried it all. The file I created is in the save directory I was working in from the terminal, so I tried

logfile=bzfslog

and I tried

logfile=/home/death/logfile.txt

and it didn't like it. So... I took out the logfile flag from the command. Maybe I was doing something wrong. THought?

But if anyone wants - get some people up, go to Planet MoFo, and everyone kill everyone else with geno and see if you can't make it crash. The exact same configuration for the server has crashed when testing this, so I dare someone to get a gang together and go to Planet MoFO and wreak havoc.

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 9:17 pm

ok...

I did valgrind just like I stated. I did recompile todays CVS with --enable-debug during configure...... re-made all my plugins. In this instance, I loaded ONLY rogue genocide.

The core dump still took place, but when I tried to do a backtrace like I've done in the past, it said the core didnt match, and to go to hell. There was not a file dumped like DTRemenak described would happen.... the bzfslog.pidXXXXX file... not there.

I'm sure I'm doing SOMETHING wrong, although valgrind says errors.

Code: Select all

Player connect time: 141.468277
Player messages in: al:12(4) en:1(251) kl:8(11) ps:449(31) sb:28(47) se:8(9) total:506(15694) max msgs/bytes per second: 23/713
Player messages out: ac:1(5) al:67(21) ap:4(175) df:6(62) fu:13(63) gf:7(62) kl:59(12) mg:16(82) pb:4(21) ps:2379(34) rp:2(5) sb:263(47) sc:59(19) se:46(9) sv:3(1014) tp:2(9) tu:63(21) total:2994(96911) max msgs/bytes per second: 53/4191
Player connect time: 141.443608
Player messages in: al:13(4) en:1(251) kl:10(11) ps:444(31) sb:66(47) se:10(9) total:544(17369) max msgs/bytes per second: 23/713
Player messages out: ac:1(5) al:67(21) ap:3(175) df:6(62) fu:13(63) gf:7(62) kl:59(12) mg:15(82) pb:3(21) ps:2383(34) rp:2(5) sb:263(47) sc:59(19) se:46(9) sv:3(1014) tp:2(9) tu:62(21) total:2994(96790) max msgs/bytes per second: 49/3988
Player [0] accept() from 69.28.129.166:37020 on 6
Player [0] submitted reverse resolve query
Player  [0] removed at 2005-08-28 17:07:29: Disconnected
Queuing ADD message to list server
Player [0] accept() from 69.28.129.166:37021 on 6
Player [0] submitted reverse resolve query
Player  [0] removed at 2005-08-28 17:07:29: Disconnected
Player [0] accept() from 192.168.1.1:53444 on 6
Player [0] submitted reverse resolve query
Player  [0] inbound UDP up 192.168.1.1:53444 actual 53444
Player  [0] outbound UDP up
Player [0] failed to resolve: error 4
Player Darth Vader [0] sent version string: 2.0.3.20050821-DEVEL-linux-SDL
Time elapsed: 0.004126 seconds
Player Darth Vader [0] has joined from 192.168.1.1 at 2005-08-28 17:07:34 with token "1306573705"
Queuing ADD message to list server
Player [1] accept() from 69.28.129.166:37026 on 7
Player [1] submitted reverse resolve query
Player  [1] removed at 2005-08-28 17:07:34: Disconnected
Got: TOKGOOD: Darth Vader:MoFo.Admin:MoFo.Cop
Identify DARTH VADER
Queuing ADD message to list server
Player [1] accept() from 69.28.129.166:37027 on 7
Player [1] submitted reverse resolve query
Player  [1] removed at 2005-08-28 17:07:34: Disconnected
Player [1] accept() from 201.26.95.14:1411 on 7
Player [1] submitted reverse resolve query
==10435==
==10435== Invalid read of size 1
==10435==    at 0x1B906116: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x8105B71: bz_killPlayer(int, bool, int, char const*) (bzfsAPI.cxx:1143)
==10435==    by 0x1B906136: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x818E5BB: handleCommand(int, void const*, bool) (bzfs.cxx:3126)
==10435==    by 0x818FFFE: handleTcp(NetHandler&, int, RxStatus) (bzfs.cxx:3653)
==10435==    by 0x81471C4: GameKeeper::Player::handleTcpPacket(fd_set*) (GameKeeper.cxx:309)
==10435==    by 0x8195EEA: main (bzfs.cxx:4803)
==10435==  Address 0x2C is not stack'd, malloc'd or (recently) free'd
==10435==
==10435== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==10435==  Access not within mapped region at address 0x2C
==10435==    at 0x1B906116: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x8105B71: bz_killPlayer(int, bool, int, char const*) (bzfsAPI.cxx:1143)
==10435==    by 0x1B906136: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x818E5BB: handleCommand(int, void const*, bool) (bzfs.cxx:3126)
==10435==    by 0x818FFFE: handleTcp(NetHandler&, int, RxStatus) (bzfs.cxx:3653)
==10435==    by 0x81471C4: GameKeeper::Player::handleTcpPacket(fd_set*) (GameKeeper.cxx:309)
==10435==    by 0x8195EEA: main (bzfs.cxx:4803)
==10435==
==10435== ERROR SUMMARY: 79 errors from 12 contexts (suppressed: 40 from 2)
==10435== malloc/free: in use at exit: 278648 bytes in 991 blocks.
==10435== malloc/free: 18114 allocs, 17123 frees, 2556961 bytes allocated.
==10435== For counts of detected errors, rerun with: -v
==10435== searching for pointers to 991 not-freed blocks.
==10435== checked 552624 bytes.
==10435==
==10435==
==10435== 28 bytes in 1 blocks are possibly lost in loss record 68 of 135
==10435==    at 0x1B9000BA: operator new[](unsigned) (in /usr/lib/valgrind/vgpreload_memcheck.so)
==10435==    by 0x81F018B: FlagType::FlagType(char const*, char const*, FlagEndurance, ShotType, FlagQuality, TeamColor, char const*) (Flag.h:126)
==10435==    by 0x81EDD14: Flags::init() (Flag.cxx:86)
==10435==    by 0x8191AEB: main (bzfs.cxx:3947)
==10435==
==10435==
==10435== 104 (64 direct, 40 indirect) bytes in 1 blocks are definitely lost in loss record 90 of 135
==10435==    at 0x1B8FFC78: operator new(unsigned) (in /usr/lib/valgrind/vgpreload_memcheck.so)
==10435==    by 0x810379B: bz_getPlayerByIndex(int) (bzfsAPI.cxx:668)
==10435==    by 0x1B906110: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x8105B71: bz_killPlayer(int, bool, int, char const*) (bzfsAPI.cxx:1143)
==10435==    by 0x1B906136: RogueDeathHandler::process(bz_EventData*) (in /home/death/bzflag/plugins/RogueGenocide/RogueGenocide.so)
==10435==    by 0x816A856: WorldEventManager::callEvents(bz_eEventType, bz_EventData*) (WorldEventManager.cxx:97)
==10435==    by 0x818B468: playerKilled(int, int, int, short, FlagType const*, int, bool) (bzfs.cxx:2261)
==10435==    by 0x818E5BB: handleCommand(int, void const*, bool) (bzfs.cxx:3126)
==10435==    by 0x818FFFE: handleTcp(NetHandler&, int, RxStatus) (bzfs.cxx:3653)
==10435==    by 0x81471C4: GameKeeper::Player::handleTcpPacket(fd_set*) (GameKeeper.cxx:309)
==10435==
==10435==
==10435== 1844 bytes in 1 blocks are possibly lost in loss record 119 of 135
==10435==    at 0x1B9000BA: operator new[](unsigned) (in /usr/lib/valgrind/vgpreload_memcheck.so)
==10435==    by 0x81452C5: FlagInfo::setSize(int) (FlagInfo.cxx:77)
==10435==    by 0x8129EE4: parse(int, char**, CmdLineOptions&, bool) (CmdLineOptions.cxx:1251)
==10435==    by 0x819235E: main (bzfs.cxx:3968)
==10435==
==10435==
==10435== 2000 bytes in 3 blocks are possibly lost in loss record 120 of 135
==10435==    at 0x1B8FFC78: operator new(unsigned) (in /usr/lib/valgrind/vgpreload_memcheck.so)
==10435==    by 0x1B9AA1A4: std::__default_alloc_template<true, 0>::allocate(unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9ACAA7: std::string::_Rep::_S_create(unsigned, std::allocator<char> const&) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9AD13B: std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9AD1E5: std::string::reserve(unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9AE9B2: std::string::append(char const*, unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9AEA34: std::string::operator+=(char const*) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x812491D: parse(int, char**, CmdLineOptions&, bool) (CmdLineOptions.cxx:560)
==10435==    by 0x8124F42: parse(int, char**, CmdLineOptions&, bool) (CmdLineOptions.cxx:612)
==10435==    by 0x819235E: main (bzfs.cxx:3968)
==10435==
==10435==
==10435== 30864 bytes in 10 blocks are possibly lost in loss record 132 of 135
==10435==    at 0x1B8FFC78: operator new(unsigned) (in /usr/lib/valgrind/vgpreload_memcheck.so)
==10435==    by 0x1B9A9E28: std::__default_alloc_template<true, 0>::_S_chunk_alloc(unsigned, int&) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9A9F1C: std::__default_alloc_template<true, 0>::_S_refill(unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9AA236: std::__default_alloc_template<true, 0>::allocate(unsigned) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9ACAA7: std::string::_Rep::_S_create(unsigned, std::allocator<char> const&) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9ACB0F: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x1B9ACC35: std::string::string(char const*, std::allocator<char> const&) (in /usr/lib/libstdc++.so.5.0.7)
==10435==    by 0x8219B18: __static_initialization_and_destruction_0(int, int) (BundleMgr.cxx:29)
==10435==    by 0x8219B98: _GLOBAL__I__ZN9BundleMgr13currentBundleE (BundleMgr.cxx:117)
==10435==    by 0x8225D68: (within /home/death/bzflag/bin/bzfs)
==10435==    by 0x80F8150: (within /home/death/bzflag/bin/bzfs)
==10435==    by 0x8225CD0: __libc_csu_init (elf-init.c:77)
==10435==
==10435== LEAK SUMMARY:
==10435==    definitely lost: 64 bytes in 1 blocks.
==10435==    indirectly lost: 40 bytes in 6 blocks.
==10435==      possibly lost: 34736 bytes in 15 blocks.
==10435==    still reachable: 243808 bytes in 969 blocks.
==10435==         suppressed: 0 bytes in 0 blocks.
==10435== Reachable blocks (those to which a pointer was found) are not shown.
==10435== To see them, rerun with: --show-reachable=yes
Segmentation fault (core dumped)
death@linux:~> gdb --core=/home/death/core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Core was generated by `valgrind --gen-suppressions=no --db-attach=no --leak-check=yes ./bzfs -conf FFA'.
Program terminated with signal 11, Segmentation fault.
#0  0xb0021ae8 in ?? ()
(gdb) file /home/death/bzflag/bin/bzfs
warning: core file may not match specified executable file.
Reading symbols from /home/death/bzflag/bin/bzfs...done.
Using host libthread_db library "/lib/tls/libthread_db.so.1".
(gdb) bt
#0  0xb0021ae8 in ?? ()
(gdb)

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 10:05 pm

Woot! I know I'm not supposed to be "happy" the server crashed, but here it is - the file DTRemenak said valgrind would create.

I had to compress it, as the board would not allow the extension (which is a bunch of numbers) (you guys ought to fix that)

I_Died_Once · Post by **I_Died_Once** » Sun Aug 28, 2005 10:31 pm

Yay! Progress! THe posted dialogue might not be helpful, but here it is. I was on with alot of bots, someone else joined - then i rejoined with no bots, and the first geno pop crashed it.

Woot.

Code: Select all

layer Darth Vader [0] has joined from 192.168.1.1 at 2005-08-28 18:18:17 with token "540267851"
Queuing ADD message to list server
Player [1] accept() from 69.28.129.166:41192 on 7
Player [1] submitted reverse resolve query
Player  [1] removed at 2005-08-28 18:18:17: Disconnected
Got: TOKGOOD: Darth Vader:MoFo.Admin:MoFo.Cop
Identify DARTH VADER
Queuing ADD message to list server
Player [1] accept() from 69.28.129.166:41194 on 7
Player [1] submitted reverse resolve query
Player  [1] removed at 2005-08-28 18:18:18: Disconnected
Player Darth Vader [0] -> All: hit me
Player Lithium 2 [7] removed at 2005-08-28 18:18:23: left
Queuing ADD message to list server
Player connect time: 50.945880
Player messages in: al:3(4) df:2(16) en:1(251) ex:1(4) kl:1(11) nf:1(98) of:1(5) og:1(4) pi:5(6) ps:68(31) sb:5(47) se:1(9) wh:1(4) ws:1(4) total:92(2807) max msgs/bytes per second: 11/405
Player messages out: ac:1(5) al:25(21) ap:9(175) df:2(62) fu:6(975) gf:2(62) gs:1(32) kl:24(12) mg:17(82) nf:1(4) of:1(4) og:1(4) pb:2(21) pi:5(6) ps:567(31) rp:6(5) sb:77(47) sc:24(19) se:16(9) sv:3(1014) tu:33(45) wh:1(38) total:824(30287) max msgs/bytes per second: 47/6651
Player [1] accept() from 24.15.179.85:2561 on 7
Player [1] submitted reverse resolve query
Player [1] resolved to c-24-15-179-85.hsd1.il.comcast.net
Player  [1] inbound UDP up 24.15.179.85:2561 actual 2561
Player  [1] outbound UDP up
Player [2] accept() from 69.28.129.166:41201 on 8
Player [2] submitted reverse resolve query
Player  [2] removed at 2005-08-28 18:18:24: Disconnected
Player Lithium 2 [1] sent version string: 2.0.2.20050318-STABLE-W32VC71
Time elapsed: 0.033827 seconds
Player Lithium 2 [1] has joined from 24.15.179.85 at 2005-08-28 18:18:24 with token ""
Queuing ADD message to list server
Player [2] accept() from 69.28.129.166:41202 on 8
Player [2] submitted reverse resolve query
Player  [2] removed at 2005-08-28 18:18:24: Disconnected
Segmentation fault (core dumped)
death@linux:~>