EGI SVG Advisories

Advisory-SVG-2016-10837

Title:   EGI SVG Advisory [TLP:WHITE] 'High' risk Configuration issue
         concerning dCache [EGI-SVG-2016-10837]

Date:    2016-05-25
Updated:

Affected Software and Risk
==========================

'HIGH' risk configuration issue concerning dCache

Package :dCache

Actions Required/Recommended
============================

Sites running dCache should check the configuration of the firewalls against
information in [R 1] and modify if necessary.  Sites may also follow the
instructions below.

Affected software Details.
==========================

This is not a software vulnerability, but some sites running dCache have been
found to have an insecure configuration (and have already been informed) but we
recommend that all sites running dCache check.

A badly configured site could allow the execution of arbitrary commands.

Component installation information
==================================

N/A

TLP and URL
===========

** WHITE information - Unlimited distribution                               **
** See https://go.egi.eu/tlp for distribution restrictions **

URL:   https://advisories.egi.eu/Advisory-SVG-2016-10837

Minor updates may be made without re-distribution to the sites

If you find or become aware of a vulnerability which is relevant to EGI you may
report it by e-mail to

report-vulnerability at egi.eu

the EGI Software Vulnerability Group will take a look.


Full description and instructions provided by dCache
====================================================

Executive summary:

Sites that have not correctly configured a server-local firewall restricting
UDP and TCP port 11111 traffic on the server hosting dCacheDomain are allowing
an attacker to execute arbitrary commands within the dCache cluster (e.g.,
delete all data) and, by extension, run arbitrary shell commands on any server
within the dCache cluster, as whichever user dCache is running under.

VULNERABILITY #1

Description:

    The dCache "LocationManager" component is responsible for
    establishing the network topology for the internal dCache
    communication.  One central "broker" server (by default, whichever
    server hosts dCacheDomain) listens for incoming UDP communication.
    This is on port 11111 by default.  All nodes within a dCache
    cluster must be able to communicate with this LocationManager.  It
    responds with information instructing a domain on how to connect
    to the rest of the dCache cluster.

    By allowing non-dCache servers to access UDP port 11111, an
    attacker is able to issue queries and commands to LocationManager.
    Careful control of these requests will allow the LocationManager
    to overwrite files on that node's local filesystem and execute
    arbitrary commands on the node running LocationManager.  The
    attacker can adjust LocationManager configuration so that domains
    that connect subsequently will relay traffic from the attacker;
    allowing her to execute arbitrary commands within dCache (e.g.,
    delete all data) and execute arbitrary shell commands on any
    server within the dCache cluster.

    Please note that site-level firewalls are only a partial solution.
    Most sites that deploy dCache also allow users from throughout the
    world to submit jobs to their HTC cluster facility.  Network
    connections from such worker nodes will likely bypass site-level
    firewalls.

    Please further note that, because UDP is a connectionless
    protocol, there is the risk of a UDP reflection attack.  If
    firewall rules fail to exclude traffic from a server running
    another UDP service (e.g., ntp, tftp) then an attacker could send
    requests to that UDP service with a faked source IP address: that
    of the server running LocationManager.  If the response from that
    server, which it will send to LocationManager, is a valid
    LocationManager command, the attacker will have successfully
    bypassed the firewall rules that forbid direct communication.


Test for vulnerability:

    Running the following command from a non-dCache-cluster machine
    checks for this vulnerability on machine hosting dCacheDomain
    ("dcache-node.example.org" here) :

        echo "ls setup" | nc -q1 -u dcache-node.example.org 11111

    Typical output if vulnerable:

        paul@sparkplug:~$ echo "ls setup" | nc -u localhost 11111
        #
        # This setup was created by the LocationManager at Mon Mar 14 10:29:36
        # CET 2016
        #
        define dCacheDomain
        listen dCacheDomain
        define *
        defaultroute * dCacheDomain
        connect * dCacheDomain
        paul@sparkplug:~$

    Typical output if not vulnerable:

        paul@sparkplug:~$ echo "ls setup" | nc -u localhost 11111
        paul@sparkplug:~$


Limiting factors:

    Commands are only executed on nodes within the dCache cluster.

    By default, RPM- and deb- based dCache deployments run as the
    unprivileged user "dcache", limiting the damage on the server.


Mitigation:

    The admin configures a machine-local firewall (e.g., iptables) so
    that it only accepts UDP port 11111 traffic from machines within
    the dCache cluster.

    A valid configuration to allow three machines to communicate with
    LocationManager while preventing attempts from all other machines:

        iptables -A INPUT -p udp --dport 11111 -s 203.0.113.10 -j ACCEPT
        iptables -A INPUT -p udp --dport 11111 -s 203.0.113.11 -j ACCEPT
        iptables -A INPUT -p udp --dport 11111 -s 203.0.113.12 -j ACCEPT
        iptables -A INPUT -p udp --dport 11111 -j REJECT

    A configuration that allows members of a subnet to access
    LocationManager while preventing attempts from all other machines:

        iptables -A INPUT -p udp --dport 11111 -s 203.0.113.0/24 -j ACCEPT
        iptables -A INPUT -p udp --dport 11111 -j REJECT


VULNERABILITY #2

Description:

    Domains pass messages between themselves using TCP connections.
    By default, these TCP connections use port 11111.

    By allowing non-dCache nodes to connect to TCP port 11111, an
    attacker can issue arbitrary internal dCache commands on any node
    within the dCache cluster; this includes the ability to delete any
    data stored in dCache.  Additionally, these dCache commands allow
    the attacker to issue arbitrary external shell commands on the
    servers.

    Please note that site-level firewalls that block port 11111 are
    only a partial solution.  Most sites that deploy dCache also allow
    users from throughout the world to submit jobs to their HTC
    cluster facility.  Network connections from such worker nodes will
    likely bypass site-level firewalls.

Test for vulnerability:

    Running the following command from a non-dCache-cluster machine
    checks for this vulnerability on machine hosting dCacheDomain
    ("dcache-node.example.org" here) :

        echo "" | nc -q1 core.dcache-cluster.example.org 11111

    Typical output if vulnerable:

        paul@sparkplug:~$ echo "" | nc localhost 11111
        ..sr dmg.cells.nucleus.CellDomainInfo.g$5.L
                _domainNametLjava/lang/String;_versionq~xpt
                        dCacheDomaint2.15.2-SNAPSHOT
        paul@sparkplug:~$

    Typical output if not vulnerable:

        paul@sparkplug:~$ echo "" | nc localhost 11111
        paul@sparkplug:~$


Limiting factors:

    The attacker can only run commands as whichever user the dCache
    JVM is running.  By default, RPM- and deb- based dCache
    deployments run as the unprivileged user "dcache", limited the
    operations that the attacker can undertake.


Mitigation:

    The admin configures a local firewall (e.g., iptables) on the
    machine running dCacheDomain so that it only allows TCP port 11111
    connections from machines within the dCache cluster.

    A valid configuration to allow three machines to join the dCache
    cluster while preventing attempts from all other machines:

        iptables -A INPUT -p tcp --dport 11111 -s 203.0.113.10 -j ACCEPT
        iptables -A INPUT -p tcp --dport 11111 -s 203.0.113.11 -j ACCEPT
        iptables -A INPUT -p tcp --dport 11111 -s 203.0.113.12 -j ACCEPT
        iptables -A INPUT -p tcp --dport 11111 -j REJECT

    A configuration that allows members of a subnet to join the
    dCache cluster while preventing attempts from all other machines:

        iptables -A INPUT -p tcp --dport 11111 -s 203.0.113.0/24 -j ACCEPT
        iptables -A INPUT -p tcp --dport 11111 -j REJECT

Credit
======

SVG was alerted to this problem by Paul Millar from the dCache team

References
==========

[R 1] https://www.dcache.org/manuals/Book-2.10/start/in-securing.shtml

Timeline
========
Yyyy-mm-dd  [EGI-SVG-2016-10837]

2016-04-07 SVG alerted to this issue by Paul Millar and information available
2016-04-07 Acknowledgement from the EGI SVG to the reporter
2016-04-21 EGI SVG Risk Assessment completed
2016-      Further discussion on how to approach this.
2016-05-09 Sites known to be vulnerable ticketed by CSIRT
2016-05-25 All sites running dCache are recommended to check their
           configuration
2016-06-08 Advisory placed on Wiki