On June 20 and 21, 2022, the fourth annual community meeting of Czech and Slovak network administrators, CSNOG 2022, took place. On the first day, the main topics discussed were problematic network traffic, routing loops and Internet reliability. On the second day, the main topics discussed were automated network management and content blocking on the Internet.
The organizers of the CSNOG event are CZ.NIC, NIX.CZ and CESNET. The program of CSNOG is coordinated by the Program Committee.
CSNOG 2022 in numbers:
105 participants, mainly from the Czech and Slovakia
16 lecturers
16 talks
1 tutorial
6 partners: GOLD: Flexoptix GmbH, Nokia, PROFiber Networking CZ s.r.o.; SILVER: RIPE NCC, Internet Society, ICANN
Presentations and videos from this year's CSNOG 2022 are available on the event website the tab is called Programme. A comprehensive summary of the most interesting individual presentations was written by Petr Krčmář, who is a member of the Program Committee.
June 20
Vojtěch Bumba: IP Address Reputation Database
The Czech Republic is not among the most successful countries in the field of cyber-attacks, and things are getting worse. Traffic analysis is becoming more and more difficult and will probably not be possible at all in the future. The solution is to use IP addresses that can never be hidden. The Kernun Central Station tool seeks to build a database of troublesome addresses that can later become the heart of a security solution. A firewall regularly downloads data to see how the Czech Internet is doing and can block all sorts of problems. Kernun has teamed up with a number of companies and organizations that monitor Internet traffic, preserving the option of human intervention so that records can be removed or added. The data collected needs to be handled in some way to create a database. Each address is given a hazard rating and monitored within a time frame. If an address is recorded once, it gets a certain rating, and if it reappears soon, the rating is multiplied. Each device from which data is collected also has a reputation and is assigned a different importance. Public services such as Google's DNS resolver cause problems. Similarly, public hosting services cause problems, with a single compromised website getting its entire IP address blacklisted. This has been solved by distinguishing hazard routes, i.e., whether or not the client has already communicated with the IP address.
Michal Hrušecký: Turris Sentinel without Turris routers
Sentinel is focused on external traffic and one of its goals is to create a list of malicious addresses that are the source of attacks. The data comes from so-called minipots, which are tiny honeypots for FTP, SMTP, HTTP and Telnet protocols. A potential attacker has the option of logging in. If they try to do so, the login credentials used are recorded and the attacker is disconnected. Such a minimalistic solution helps avert the security problems that would inevitably arise in a more complex solution. The data is then used to generate a list of addresses that behave unsafely. The goal is to get the addresses on the lists very quickly, but also to remove them quickly when they start behaving. The data collection is done on Turris routers because it is difficult to ensure the trustworthiness of the data source, but we are working on making it possible to join without a Turris router. The current solution uses the Turris router serial number to identify specific users. In the case of public data collection, the system will pretend that each user has one router. Everyone will generate their own serial number and send it to CZ.NIC. In the first phase, cooperation will be offered to large companies that already have a unique identification and can provide trustworthy data from multiple IP addresses. It will be more difficult with individual users, as each may have hundreds of e-mail addresses and login credentials. At the moment we are working on making this available to the general public. (Video ENG)
Shannon Weyrick: processing network data at the edge of the network
NS1 runs authoritative servers in 26 locations around the world, collecting a lot of network data. It keeps all the data at the edge of the infrastructure, where it processes it. The tool used to do this is called Orb. This open-source tool processes small amounts of data and then allows the results to be viewed on-site or sent to a central storage facility. The result is reports that allow you to track various metrics, from IP addresses to DNS query types. The entire solution consists of agents located at the edge of the network and a central control hub that allows you to orchestrate the agents and keep track of everything happening on the network. It allows you to create rules on what data you want to collect and process. The main work is done by the agents themselves, whose purpose is to split the massive amount of data into individual streams as quickly as possible. Afterwards, this data is analyzed in real time and the aggregated data can be passed on to the central office. In this way, a large amount of information can be monitored and this processing can be supplemented by additional modules. (Video ENG)
Marian Rychtecký: Step-by-step migration to VxLAN / EVPN
In 2019, it was clear that data flows were increasing and 400GE ports would become a necessity. At the same time, NIX.CZ needed to be connected in three locations. This was to create a network with a triangular topology, but L2 networks don't like circles. In addition, the Nexus 7010 switches used in the node were nearing the end of their service life. So the question on the table was whether to stay with the large-chassis solutions or switch to conventional small switches. It is not possible to combine the two. In the end, however, it was decided to transition to standalone 1U modules. It was also necessary to decide whether to organize the network using Flat and learn or EVPN. Port security proved to be essential, as it must provide protection against theft, spoofing or MAC address collisions. This happens on a regular basis, for example loops occur several times a day. In addition to port security, it was also necessary to manage MAC addresses according to different VLANs. The peering node supports only a fixed number of MAC addresses per port, today it is two, in the future it will be only one. The port learns the first MAC address used and enables it. If you use a different one, it will drop it. If a colliding MAC address came from another direction, EVPN would move it, but port security would block the operation and the data flow to the address would cease. Rectifying it requires manual intervention, because it won't fix itself. NIX.CZ solved this with a script that watches for addresses blocked in this way and removes them from the table. The advantage of the new solution is that it is much easier to manage and it is possible, for example, to redirect internal traffic due to maintenance very easily. The disadvantage is that it is a relatively new technology that is still maturing and needs to be learned by everyone.
Alexander Zubkov: routing cycles
A routing cycle is a situation where a packet is routed in an infinite loop. There is a safeguard against this in the IP protocol and each packet has a TTL value set, i.e. the number of routers it can pass through. However, this value can be modified and set up to 255. Cycles can be checked, for example, by using the traceroute tool, which allows the entire path to be mapped. Most of these problems occur only for a very short period of time, but in case of a misconfiguration, for example, long cycles can occur. Why should we care about cycles at all? They can cause major traffic spikes, which can then be exploited in a DDoS attack on your links. In some cases, such a loop can even be exploited as a source of attacks. Details can be researched at radar.qrator.net, where you can enter an autonomous system number.
Alexander Kozlov: Measuring Internet reliability in 2022
One of the criteria is the stability of the national internet, which is measured according to the centralization of networks in each country. In 2021, the most critical operator has been changed to ASN 39392, which is SuperNetwork. With this, the country's internet stability deteriorated as it dropped from 21st to 37th place in the global ranking. In fact, SuperNetwork has increased its share of connected networks to 8.9%. Any such centralization is not very good for the Internet. The situation is similar for IPv6, where the largest operator has changed and the situation has also worsened. In terms of reverse records (PTRs), the most critical provider is O2, which controls more than 27% of the country's records. In Slovakia it is SK Telekom, which controls as much as 41% of records. If I were connected to just one of them, I'd get an extra line somewhere else. If it goes down, it will be really bad. But these statistics may be affected by the fact that some providers give clients public addresses and others use CG NAT.
Zbyněk Kocur: Objective mobile network measurement
When mobile network coverage is verified, parameters are usually measured at the physical layer. This is done by the Czech Telecommunication Office, which is now gradually going higher. It is possible to do measurements at higher layers, with the highest one being TCP/IP. Variation then depends on where the measurements are taken. Firstly, it is possible to measure with a static test, which is repeatable. But this does not match the way mobile networks are used today. The nomadic test is performed in several different places. In the case of TCP/IP, this may be due to the downlink being pass-through but the uplink being congested. In such a situation, a disconnection may occur. TCP is designed to operate independently of problems at the physical layer. This works well for Ethernet where we can disconnect the cable. But with wireless connections, the line goes dead and the data doesn't flow. Then the timeouts kick in, with TCP waiting up to 30 seconds. This is quite common on cellular networks: when you are driving on the D1 motorway, the connection may disappear for several minutes. Then the connection may not be re-established at higher layers, which affects the measurements. TCP Cubic algorithm is gradually being replaced by the more modern BBR algorithm. Cubic tries to use the maximum capacity even at higher RTT, taking into account competing flows as well. In the case of BBR, detection packets are sent to the network, according to which the flow is controlled. Even if the error rate increases, BBR does not care and as long as the RTT remains unchanged, it is able to keep adding and adding. Cubic slows down significantly already at 0.01% loss rate, so BBR can kill it very efficiently. BBR can adapt much better to the traffic conditions in the mobile network.
Jaromír Novák: radio spectrum and access to it after 2022
The radio spectrum is not a simple matter from the technical point of view, and even more confusion arises when you try to regulate it legally. These days, the spectrum is the key to doing business in electronic communications and not only there. It is a limited but inexhaustible resource that is an asset to the International Telecommunications Union (ITU), which is trying to make services work across the globe. The ITU holds a regular forum every four years to try to coordinate the use of the spectrum. At this level, Europe is represented by CEPT, with individual countries implementing the agreed rules. The main mechanisms for the spectrum use are: general authorization, individual authorization for the spectrum use, individual authorization for experimental purposes and allocation. In some of these schemes, the user pays a regular annual fee and is then assured that no one else can use its frequencies. The Czech Telecommunication Office is preparing new objectives for the next period: the EU common policy and proposals for the radio spectrum, terrestrial TV broadcasting and bands below 700 MHz (further TV band cuts), development of terrestrial digital audio broadcasting DAB (band III), millimeter bands for 5G (26 GHz), extension of the upper 6 GHz for shared access networks, conditions of use of frequencies in the 410/420 MHz and 450/460 MHz bands.
June 21
Tomáš Kubina: NETCONF vs. gNMI
The NETCONF protocol has been around since 2006 and its role is to provide an application interface for network devices to be easily managed. It is about retrieving and configuring the current state of a network element. The good thing about it is that we don't have to get all the data at once, as it offers filtering. We can specify what we want and get the requested information. It is based on the remote procedure call (RPC) principle. The gNMI protocol was created in 2015 and is backed by the OpenConfig initiative. Users of network devices agreed to create a protocol that would be usable across network elements. It is built on the gRPC framework, over which a specification for the needs of the networking world has been prepared. Like NETCONF, gNMI is intended to provide an interface for network element management. The advantage of gNMI is that it is more easily encodable into binary format, which has a positive impact on the speed and amount of data transfer. If you use gNMI, up to four times less data is transferred than with NETCONF. The more data that is transferred, the more the transfer efficiency is affected. This is similar for response time. The HTTP/2 protocol including TLS encryption is used here for message transfer.
Alexander Zubkov: automatic update of prefix lists
Qrator Labs runs dozens of routing nodes on Linux using the BIRD daemon. The data for filtering is generated automatically using Ansible. It is necessary to download AS sets, generate a configuration for BIRD, and send it a command to use the new configuration. The whole solution is built on three components: Ansible, Plag HTTP and updatefilter scripts that do various jobs directly on the servers. You can set explicit values in the prefix list and also dynamic components that are automatically configured. The lists can be retrieved using bgpq4, which caches the information so that each server does not overwhelm the registrar's servers. We have a private HTTP API implemented using Nginx where you can request prefixes.
Jaroslav Zdeněk: Managing networks by students, for students
The network is very large, there are 120 access switches, with two C9500 switches at the core. Primary connectivity is via Czech Technical University to the CESNET network. The logical topology of the network consists of about 90 VLANs operated by one Catalyst 9500 router. The network is built on public IPv4 addresses and also supports IPv6. Members have fixed IPv4 and IPv6 addresses with identification by MAC address. IPv6 addresses are assigned using DHCPy6d, which means that the server must be in all user VLANs. A more modern solution is RFC 6939, which allows the MAC address to be included in the request. We have tried this, and among DHCP servers, it is supported by at least ISC Kea. The Wi-Fi network uses private IPv4 and public IPv6 addresses. Because of Android, DHCPv6 cannot be used, so addresses have to be allocated using SLAAC and 802.1X takes care of the order in the network. We have tested that as soon as a user sets up a static IPv6 address and the first packet arrives, the information flies immediately to the RADIUS server. There are also more and more users who want to connect various IoT devices to Wi-Fi, which Strahov has solved with MPSK and iPSK.
Alexander Isavnin: to block or not to block?
The Russian Federation has a great deal of experience in blocking inconvenient content. This year we celebrate ten years since the implementation of the new blocking laws. In the beginning, the public was prepared for content blocking by cases such as the Blue Whale, when citizens began to ask who would protect children from such threats. Everything was in place to block content in the name of protecting and keeping our children safe. The first step was to label the content as harmful, and the regulatory authority Roskomnadzor was to try to contact the operator and demand its removal. The topics subject to blocking are gradually increasing: copyright infringement, terrorist propaganda, much more information dangerous to children, online alcohol sales, gambling and more. Russia has another agenda it is trying to pursue with this. Eventually, the first terrorists to be blocked were Kasparov and Navalny. But the blocks targeting them are also not effective. The main effect of the blocks is that they impose obligations on the operators who are now responsible for the blocking, and who are being given new targets: search engines, news aggregators, various VPNs, communicators and so on. So the Internet in Russia is heavily regulated by the operators. The networks are strictly monitored and face heavy fines, while it is not specified exactly how to implement the blocking and how the whole system is supposed to work. The blocking doesn't work from a technical point of view either; the really problematic sources can be swiftly moved elsewhere. The result is therefore only collateral damage, all sorts of failures and a heavy load on the network. The expert technical community tries to explain this but gets ignored.
Dmitry Kohmanyuk: hosting the Ukrainian domain in CZ.NIC
The Ukrainian domain showed signs of a crisis as early as January 15, when a large DDoS attack targeted authoritative servers. It caused a failure of part of the internal infrastructure, which stopped updates in the zone. The domain operators contacted CZ.NIC to see if it would provide servers to host the Ukrainian domain. Communication was very fast through various online services. Regular online meetings were held daily to discuss the technical details of cooperation. The Czech team was very flexible and helpful. The first servers were configured within three days. During the cooperation we also learned a lot of new things. CZ.NIC is now helping with running the basic infrastructure and DNS, but other companies such as CloudNS, Cloudflare, Gransy, Netnod and others are also involved. I would like to thank all the members, staff and management of CZ.NIC.
Ondřej Filip: How close are we to the Internet split?
What holds the Internet together? Ondřej Filip listed five things in the order of importance: uniform standards, uniform addressing system, uniform routing, uniform naming system and uniform service provision. Uniform service provision has been disrupted for a long time, usually for commercial reasons, especially by geo-blocking. But politics has also intervened by blocking or terminating various services in different countries. That layer has been disrupted a lot, and a lot of services no longer work as you would expect. But it's easy to get around that with, for example, a VPN. Another item is the single name system, which is managed by ICANN and the root zone operators. From a technical point of view, unified routing is much more important, with Tier-1 operators forming the backbone of the Internet. These network operators do not pay each other to interconnect their networks and can send data to any network in the world. This is further densified by various interconnection nodes and so on. The whole thing is very robust. This large infrastructure has already started to partially collapse, with some companies terminating contracts with the Russian operator because of the sanctions. Most of the biggest operators are from the United States or Europe, and they are private companies. The Internet is not set up by governments. At the same time, this does not mean that the state cannot influence the Internet. The unified addressing system, which is managed by the RIPE NCC in Europe, the Middle East and the post-Soviet republics, is also essential for the functioning of the Internet. It is difficult to predict whether the single Internet will split. Russia is preparing for it intensively, the West is not concerned. It would be unpleasant for both sides. It would probably be worse for the West because Russia will find another way to watch us. We may take a look at North Korea, which knows everything about us, while we do not even know when Kim Jong-un was born.
For more information about the CSNOG community meeting, those interested can visit www.csnog.eu, where they can also find an archive of previous years' meetings.
