Using systemd to restart core dump C++ program for debugging in CentOS 7

Core Dump Generation

How to write a crashing C++ program?
vi a.cpp

#include <string.h>

#include <chrono>
#include <thread>
int main() {

    std::this_thread::sleep_for(std::chrono::seconds(10)); // 10 or 120
    char* p = NULL;
    strcpy(p, "copy string to null pointer and cause crash");
    return 0;
} // g++ -g a.cpp -o a.out -std=c++11

Segmentation fault (core dumped)

Why there is no core dump file generated?
ulimit -c
0
ulimit -a | grep core
core file size          (blocks, -c) 0

How to enable core file size?

ulimit -c unlimited

ulimit -a | grep core
core file size          (blocks, -c) unlimited

What is the file size of the core dump?

417KB

./a.out

ll core.*

rw-------. 1 ricky ricky 417792 Mar 28 11:06 core.1258

Core Dump Analysis

How to setup dbg for debugging?

sudo debuginfo-install glibc-2.17-106.el7_2.8.x86_64 libgcc-4.8.5-11.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64

How to analyse the core dump file?

gdb a.out core.1258

bt

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7

Copyright (C) 2013 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /home/ricky/crash/a.out...done.

[New LWP 1258]

Core was generated by `./a.out'.

Program terminated with signal 11, Segmentation fault.

#0  0x00000000004005cb in main () at crashing.cpp:6

6           strcpy(p, "copy string to null pointer and cause crash");

(gdb) print p

$1 = 0x0

Background of systemd

What is init?
init is the default PID 1 in CentOS 6
cat /etc/*release
CentOS release 6.6
ps -ef | head -2
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0  2016 ?        00:01:03 /sbin/init

What is systemd?
systemd is the default PID 1 in CentOS 7
cat /etc/*release
CentOS Linux release 7.2.1511
ps -ef | head -2
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0  2016 ?        00:09:54 /usr/lib/systemd/systemd --system --deserialize 26

Restarting the crashing program with systemd

How to create a daemon?

sudo vim /etc/systemd/system/a.service
[Unit]
Description=service a.service here

[Service]

Type=simple

Restart=always

WorkingDirectory=/home/ricky/a/

ExecStart=/home/ricky/a/a.out

[Install]

WantedBy=multi-user.target

How to start a daemon?

sudo systemctl daemon-reload

sudo systemctl status -l a

● a.service

   Loaded: loaded (/etc/systemd/system/a.service; disabled; vendor preset: disabled)

   Active: inactive (dead)

sudo systemctl start a

sudo systemctl status -l a

● a.service

   Loaded: loaded (/etc/systemd/system/a.service; disabled; vendor preset: disabled)

   Active: active (running) since Tue 2017-03-28 11:46:36 HKT; 108ms ago

 Main PID: 2056 (a.out)

   CGroup: /system.slice/a.service

           └─2056 /home/ricky/a/a.out

ps -ef | grep a.out

How to enable core dump for systemd daemon?

sudo vi /etc/systemd/system.conf

DefaultLimitCORE=infinity
systemctl daemon-reexec



How to make a b.service that requires a.service?
sudo vim /etc/systemd/system/b.service
[Unit]
Description=service b.service here
Requires=a.service
[Service]
Type=simple
Restart=always
WorkingDirectory=/home/ricky/b/
ExecStart=/home/ricky/b/b.out
ExecStartPre=/usr/bin/date
ExecStartPost=/usr/bin/date
ExecReload=/usr/bin/date
ExecStop=/usr/bin/date
ExecStopPost=/usr/bin/date
[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl status -l b

sudo systemctl start b

sudo systemctl status -l a

sudo systemctl status -l b
ls && ps -ef | grep b.out

sudo systemctl stop a
sudo systemctl stop b

sudo systemctl start a
sudo systemctl start b

sudo systemctl status -l a
sudo systemctl status -l b

ExecStartPre=/bin/bash -c 'pwd; ulimit -c unlimited; rm ./*.log; python -c "print(123)" '?

Process: 23069 ExecStartPre=/bin/bash -c pwd; ulimit -c unlimited; rm ./*.log; python -c "print(123)"  (code=exited, status=0/SUCCESS)

Apr 07 12:41:05 pc009 bash[23069]: /home/ricky/b

Apr 07 12:41:05 pc009 bash[23069]: rm: cannot remove ‘./*.log’: No such file or directory

Apr 07 12:41:05 pc009 bash[23069]: 123

ExecStartPre=/usr/bin/sleep 5?

Yes. It will be exited with status 0/SUCCESS.

Process: 20924 ExecStartPre=/usr/bin/sleep 5 (code=exited, status=0/SUCCESS)

ExecStartPre=/usr/bin/sleep 1; /usr/bin/sleep 1; /usr/bin/sleep 3?

No. Multiple commands not allowed.

Process: 21025 ExecStartPre=/usr/bin/sleep 1; /usr/bin/sleep 1; /usr/bin/sleep 3 (code=exited, status=1/FAILURE)

ExecStartPre=rm /home/ricky/b/md/*.con ?

No. It is because rm is not absolute. You will see below error.

[/etc/systemd/system/b.service:8] Executable path is not absolute, ignoring: rm /home/ricky/b/md/*.con

ExecStartPre=/usr/bin/rm /home/ricky/b/md/*.con ?

No. It will be exited with status 1/FAILURE

Process: 20827 ExecStartPre=/usr/bin/rm /home/ricky/b/md/*.con (code=exited, status=1/FAILURE)

How to make a sleeping service?

sudo vim /etc/systemd/system/sleeping.service
[Unit]
Description=sleeping service
[Service]
Type=simple
User=ricky
Restart=always
WorkingDirectory=/home/ricky/
ExecStart=/usr/bin/sleep 10
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload && sudo systemctl restart sleeping && sudo systemctl status sleeping

● sleeping.service - sleeping service

   Loaded: loaded (/etc/systemd/system/sleeping.service; disabled; vendor preset: disabled)

   Active: active (running) since Fri 2017-04-21 16:43:19 HKT; 31ms ago

 Main PID: 28266 (sleep)

   CGroup: /system.slice/sleeping.service

           └─28266 /usr/bin/sleep 10

Apr 21 16:43:19 pc009 systemd[1]: Started sleeping service.

Apr 21 16:43:19 pc009 systemd[1]: Starting sleeping service...

abcdefghijklmnopqrstuwxyz

abcdefghijklmnopqrstuwxyz

abcdefghijklmnopqrstuwxyz

abcdefghijklmnopqrstuwxyz

abcdefghijklmnopqrstuwxyz

abcdefghijklmnopqrstuwxyz