Skip to content

debugging

gdb

This will be mostly about getting gdb to handle C++ programs.

stepping c++

The step command is one of the most useful in gdb, but its is horrible to use in c++ since it will step into each and every constructor and inlined function.

Mostly we are not interested in any code coming from /usr/include/** For instance :

changetitle
1
2
3
4
5
6
459      cmd = vehicle_ins_cmd(db, vehicle);
(gdb) s
std::shared_ptr[visit](Vehicle)::shared_ptr (this=0x7fffffffd2c0)
at /usr/include/c++/10/bits/shared_ptr.h:149
149           shared_ptr(const shared_ptr&) noexcept = default; ///< Copy constructor
(gdb)

We want to enter function vehicle_ins_cmd() and not the copy constructor of shared_ptr. Of course we could set a breakpoint on vehicle_ins_cmd but that defeats the whole point of the step operation.

gdb 7.12 has a solution for this, it can be ordered to skip certain directories from being used. Put these in your ~/.gdbinit file :

~/.gdbinit
1
2
3
skip -gfi /usr/include/c++/*/*/*
skip -gfi /usr/include/c++/*/*
skip -gfi /usr/include/c++/*

Now the debug session becomes :

~/.gdbinit
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
File(s) /usr/include/c++/*/*/* will be skipped when stepping.
File(s) /usr/include/c++/*/* will be skipped when stepping.
File(s) /usr/include/c++/* will be skipped when stepping.
Reading symbols from ./test/test...
...
459         cmd = vehicle_ins_cmd(db, vehicle);
(gdb) s
vehicle_ins_cmd[abi:cxx11](Database*, std::shared_ptr[visit](Vehicle)) (
    db=0x5555558d2a60, 
    vehicle=std::shared_ptr[visit](Vehicle) (use count 3, weak count 1) = {...})
    at test/../vehicle.cpp:431
431         std::stringstream command;

431 is indeed the first line of vehicle_ins_cmd() ;) To see the skipped directories use :

info
1
info skip

incomplete type

Still searching for a solution here....

Tryout compiling everything with -D_GLIBCXX_DEBUG

visit

Compiling with the flag results in :

debugging
1
2
3
4
Program received signal SIGSEGV, Segmentation fault.
__memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:132
132     ../sysdeps/x86_64/multiarch/memcpy-ssse3.S: No such file or directory.
(gdb)

So that's even worse, but the gdb site states that the size of containers etc change so it is likely ALL code has to be recompiled with this flag for it to work.

If you print the stack trace it will point to osrm:: functions so at least that should be recompiled.

Another thing to try is : visit It says adding debug/string instead of just string would help. But strings are printed just fine with or without this header. Anyway there is no debug/sstream to include so that won't help.

The one thing that helps is this :

solution
1
2
3
4
5
6
7
8
9
apt-get install libstdc++6-10-dbg # or find you version in aptitude
g++ -g foo.cpp 
ldd a.out # this will give a hint about what libstd++ is used
# find the debug version 
find /usr/lib | grep stdc++ | grep debug
# one of the hits should be : 
/usr/lib/x86_64-linux-gnu/debug/libstdc++.a
# g++ -g foo.cpp /usr/lib/x86_64-linux-gnu/debug/libstdc++.a
gdb ./a.out

Now printing a stringstream will dump a whole lot more ! The content is in some submember (_M_string in my case), but this is faster:

debug stringstream
1
2
p ss.str();
$2 = "abx 42 xyz 3.1415n"

For temporary compiles I added this to make backend makefile :

makefile
1
2
3
LINKDEBUG=/usr/lib/x86_64-linux-gnu/debug/libstdc++.a
debug/backend: backend.cpp debug/libbackend.a
$(CXX) $(CXXFLAGS) $(CFLAGS) $(DEBUGFLAGS) $^ -o $@ -L debug -lbackend $(LDFLAGS) $(LINKDEBUG)

Just always compile test and debug like this.

exception breakpoint

When you want to stop the debugger at the point an exception occurs, you can set a 'breakpoint' like this :

break on exception
1
2
(gdb) catch throw
(gdb) run

Very handy !!

pretty printing

Very short guide from here : visit

Download the pretty-print python file here

download pretty-print
1
2
svn co svn://gcc.gnu.org/svn/gcc/trunk/libstdc++-v3/python
mv python ~/Install/gdb_printers

Then put this inside .gdbinit :

~/.gdbinit
1
2
3
4
5
6
7
python
#sys.path.append("/usr/share/gcc-10/python/");
import sys 
sys.path.insert(0, '/home/kees/Install/gdb_printers')
from libstdcxx.v6.printers import register_libstdcxx_printers
register_libstdcxx_printers (None)
end

It should now work, but ut could also say :

ModuleNotFoundError: No module named 'libstdcxx'

If so, you should point the python code to the location of libstdcxx, in the snippet above it was already added in comments. Of course you first need to check the exact path, then uncomment. The error should now be gone.

fuzzing

In particular American Fuzzy Lob (AFL).

Installation is simply downloading the tarbal and :

fuzzing
1
2
3
4
wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
tar zxvf afl-latest.tgz
cd afl-2.52b
make

You can instrument the program by simply changing the compiler in your makefile to the compilers you just created :

instrumenting compilers
1
2
CC     = /home/kees/Install/afl-2.52b/afl-gcc
CXX    = /home/kees/Install/afl-2.52b/afl-g++

You should now be able to run afl-fuzz but...

run
1
2
3
mkdir testcases
mkdir findings
/home/kees/Install/afl-2.52b/afl-fuzz -i test_case -o findings -- ./test/test

But it will spawn this error :

error
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
afl-fuzz 2.52b by [visit](lcamtuf@google.com)
[+] You have 8 CPU cores and 4 runnable tasks (utilization: 50%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...

[-] Whoops, your system uses on-demand CPU frequency scaling, adjusted
   between 1367 and 3906 MHz. Unfortunately, the scaling algorithm in the
   kernel is imperfect and can miss the short-lived processes spawned by
   afl-fuzz. To keep things moving, run these commands as root:

   cd /sys/devices/system/cpu
   echo performance | tee cpu*/cpufreq/scaling_governor

   You can later go back to the original state by replacing 'performance' with
   'ondemand'. If you don't want to change the settings, set AFL_SKIP_CPUFREQ
   to make afl-fuzz skip this check - but expect some performance drop.

[-] PROGRAM ABORT : Suboptimal CPU scaling governor
         Location : check_cpu_governor(), afl-fuzz.c:7337

Which explains in detail how to solve this you should note down the way to turn on scaling again.

libfuzzer

To be done

See here for a good start : visit

address sanitizer

This is actually a google project but it is closely related to the clang suite where libfuzzer

Both are also activated in a similar way :

sanitizer
1
2
clang-9 -fsanitize=fuzzer test.c
clang-9 -fsanitize=address test.c

You can also specify both comma separated.

Here is an example that caused problems in detail :

example problem
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
char *normalize_zip(const char *src)
{
   if (!src) return NULL;

   char *dst= strdup(src);
   char *result = dst;

   while (*src && len--) {
      if (isspace(*src)) src++;
      else *dst++ = toupper(*src++);
   }
   *dst = '0';

   return result;
}

This seems like a normal function with some safe guards but actually the first test case with libfuzzer crashes it.

output
1
SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/kees/projects/san/a.out+0x431128) in strdup

It took some time to see how fuzzer works. Here is a complete walk trough. The function above is tested with this 'wrapper' for libfuzzer.

I created a directory structure for fuzzer tests :

directory structure
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
fuzzer/
└── util
    └── normalize_zip
        ├── corpus
        │   ├── 03d5f51f38198db51103d878a552a4611eb4e07f
        │   ├── 04038448ce858e0cd698944e27c8532a44001693
        │   ├── 043d8b913be7c47d30af08f6badd2094ccbb5c76
        │   ├── fde6174fcdfe91fa0ad584cca8914d8d05786428
        │   └── ffe0cfa36ba87655a7317cf05a10eca0b2b4f8ac
        └── fuzz.c

The corpus directory is built up for you with more and more inputs. They are just the binary input for your function that are tried. You can put your own files there to start it of on a good foot. The fuzz.c file looks like this :

fuzz.c
1
2
3
4
5
6
7
#include "../../../util.c"

int LLVMFuzzerTestOneInput(const char *data, size_t size) {
   char *res = normalize_zip(data,size);

   return 0;
}

LLVMFuzzerTestOneInput() is always the same and libfuzzer feeds it's test data through it. You can then relay that into your function to be tested.

This can be compiled and instrumented with :

instrument compile
1
2
clang-9 -fsanitize=fuzzer,address fuzz.c -o fuzz
./fuzz

When you run this it crashed and dumps crash file with length 0. If you start fuzz in the debugger and break at normalize_zip you will see :

gdb
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
where
#0  normalize_zip (src=0x53a400 [visit](fuzzer::TPC) "", len=0)
   at fuzzer/util/normalize_zip/../../../util.c:14
#1  0x00000000004fdb47 in LLVMFuzzerTestOneInput (data=0x602000000010 "276", 
   size=0) at fuzzer/util/normalize_zip/fuzz.c:4
#2  0x000000000043d6d2 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) ()
#3  0x000000000043f1ff in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector[visit](fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile) >&) ()
#4  0x000000000043f981 in fuzzer::Fuzzer::Loop(std::vector[visit](fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile) >&) ()
#5  0x000000000042d45c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) ()
#6  0x0000000000456053 in main ()

You see that LLVMFuzzerTestOneInput feeds data "276" with size 0. This explains the crash but also the crash file of 0 bytes.

If you see the report it says :

diagnose
1
0x602000000011 is located 0 bytes to the right of 1-byte region [0x602000000010,0x602000000011)

So it looks like a 1 byte region that is not null terminated. Something like this :

example
1
2
3
4
char *s = malloc(1);
s[0] = '276';

normalize_zip(s);

This would also crash on strdup(). In fact you can do nothing to test if the input is valid or not. This would be valid :

valid
1
2
3
4
5
char *s = malloc(10);
s[0] = '276';
s[9] = 0;

normalize_zip(s);

But you cannot test that from within normalize_zip (or strdup) without also specifying the valid length and using strndup !! But if we do not check the length again in the loop we again get a crash. So our final version would become :

normalize_zip
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
char *normalize_zip(const char *src, int len)
{       
   if (!src) return NULL;

   char *dst= strndup(src,len);
   char *result = dst;

   while (len> 0 && *src) { 
      if (isspace(*src)) src++;
      else *dst++ = toupper(*src++);
      len--;
   }
   *dst = '0';

   return result;
}

Note that at first the loop test was

loop
1
while (*src && len>)

But that also gave an error because you test *src first and the len second. It does mean we have to pass along len all the time, will this become a performance issue. On the other hand strlen() suffers from the same deficit so we can't use that either and maybe we should switch to GString altogether since that carries it's length around.

I will start fuzzing things and see how much effort is needed to get it concrete safe !