Hazard's stuff

Huawei OPS syslog example

— Posted by hazard @ 2023-06-29 10:54
Documentation from the Huawei site gives non-working examples and wrong names of parameters for OPS syslog logging. Below are syslog calls that actually work:
import ops

def ops_condition(_ops):
    status, err_log = _ops.syslog("Hello world", ops.INFORMATIONAL, ops.SYSLOG)
    status, err_log = _ops.syslog("Example critical error", ops.CRITICAL, ops.SYSLOG)

Example Huawei OPS route monitoring script that changes VXLAN VTEP configuration if route goes away

— Posted by hazard @ 2023-06-17 23:42
To avoid using peer-link between Huawei CloudEngine switches in PtP VXLAN environment and therefore save 4 x 100G ports, I made a Python script for Huawei OPS that changes VTEP peer IP in case route to primary VTEP disappears (e.g. primary switch fails). It was much more of an effort than it should been, due to inadequate API documentation & examples which sometimes specify wrong parameter values. Non-working Python error reporting on Huawei VRP (at least on OS release I used) didn't help either. Details below. (More)

static bearssl https client

— Posted by hazard @ 2021-12-28 21:26
From time to time I encounter a Linux system which doesn't have a proper SSL library or doesn't have one at all (e.g. embedded). For such cases I made an extremely simplistic https client using BearSSL library that can be compiled statically with appropriate libc. It is a quick hack based on client_basic.c and may not work properly in cases of unusual server responses. Example command line:
./https_client en.wikipedia.org 443 /wiki/Linux

The client returns response headers as STDERR, body as STDOUT and exit status 0 if HTTP response status is 200. To compile, untar into a folder containing compiled BearSSL library, cd https_client && ./make.sh

"Failed to install RT_NH entry" error on Juniper during routing-instance import

— Posted by hazard @ 2020-04-19 14:23
I realized why I was getting errors like this on Juniper EX4200 switch when importing routes from one routing-instance to another:
fpc2 Failed to do walk over constituent nhs (status: 1004)
/kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
fpc2 Failed to install RT_NH entry (status: 1004)
fpc2 Failed to create the RT_NH entry (status: 1004)
fpc2 RT-HAL,rt_entry_add_msg_proc,2873: rt_halp_vectors->rt_create failed
fpc2 RT-HAL,rt_entry_add_msg_proc,2933: proto ipv4,len 27 prefix X/24 nh X
fpc2 RT-HAL,rt_msg_handler,601: route process failed
I was using prefix-lists to control which routes are imported, but didn't include in the list the Local address /32 for the directly connected route I wanted to import. Seems obvious now, but until I figured it out, it wasn't :)

ISC dhcpd doesn't work for unicast renewals without ARP

— Posted by hazard @ 2018-07-14 13:29
While implementing a network isolation policy I encountered an interesting quirk of how ISC dhcpd works: for unicast renewals, it uses a Linux IP-based syscall to send DHCPACK towards an IP address, not MAC. As a result, it means that an ARP entry should exist and therefore ARP towards the client host should work. For broadcast DHCP requests, DHCP replies directly to a MAC address and ARP is not needed. I expected that DHCP server wouldn't need anything apart from ports 67/68 and isolated everything else. So, I was seeing that some devices can get DHCP while some don't, which led to quite fun troubleshooting session, especially since dhcpd was logging a successful DHCPACK response - while tcpdump was showing that no response was actually sent through the wire.

Script to automate addition of self-signed SSL certificate to Git

— Posted by hazard @ 2017-04-24 10:06
Out of the box, Git doesn't recognize self-signed SSL https repository certificates typically used in internal networks and refuses to connect: "Peer's certificate issuer has been marked as not trusted by the user". A common method is to disable certificate check altogether, which opens up possibility of MITM. A more safe solution is to add SSL certificate of your internal repository to Git's config, so that it gets checked and recognized. This reduces your vulnerability window to the initial certificate download. I made a small shell script to automate the job: it downloads the SSL certificate and adds it to Git. Credit goes to ThorSummoner for the trick to fetch the cert using OpenSSL client.
if [ ! "$1" ] ; then
    echo "Pass repository domain name as parameter (e.g. $0 git.local)"
mkdir ~/.gitcert 2>/dev/null
true | openssl s_client -connect $1:443 2>/dev/null | 
       openssl x509 -in /dev/stdin > ~/.gitcert/$1.crt
git config --global http."https://$1".sslCAInfo ~/.gitcert/$1.crt

Smokeping Mikrotik SSH plugin with VRF support

— Posted by hazard @ 2015-07-13 00:54
I have released a Smokeping plugin for Mikrotik RouterOS devices. It supports VRFs and connects via SSH. Installation instructions:
  • Download the plugin to your server.
  • Install by copying the file into lib/Smokeping/probes directory under your smokeping installation (e.g. to /opt/smokeping/lib/Smokeping/probes).
  • You might also have to install Net::OpenSSH Perl module, if it's not already installed (check by running " perl -e 'use Net::OpenSSH' ").
  • Add the following section to your smokeping config:
    + OpenSSHMikrotikPing
    packetsize = [e.g. usual MTU is 1500]
    mikrotikuser = [user]
    mikrotikpass = [pass]
    # feel free to change params below as you wish
    forks = 5
    offset = 50%         
    timeout = 15
    step = 120
  • Individual targets are configured as follows:
    ++ sample-target
      probe = OpenSSHMikrotikPing
      menu = [menu name]
      title = [title]
      host = [destination IP to ping from Mikrotik device]
      pings = [numer of ICMP pings to send, e.g. 5]
      source = [Mikrotik device to login into]
      vrf = [routing-instance name, optional]
      psource = [Mikrotik interface source IP to ping from, optional]
  • ssh to the Mikrotik device once from the commmand line from the account of the user who is running smokeping (su -s /bin/sh [username]). On the first connect ssh will ask to add the new host to its known_hosts file, confirm it. Otherwise Smokeping will fail to login as the ssh key of your Mikrotik box is not in the known_hosts file.

bash binaries with ShellShock vulnaribility patch for old Red Hat Linux systems

— Posted by hazard @ 2014-09-25 19:06
** UPDATED SEP 26 2014 FOR CVE-2014-7169 **

Some of us are unlucky enough to run older Linux systems (CentOS 4 and older) and need to fix bash "ShellSock" environment code injection vulnerability.

To make the job easier, you can grab my CentOS 4.x i386 RPM/SRC RPM , as well as Red Hat Linux 6.2 (circa 2000!) RPM/SRC RPM, which may work on older systems as well, such as RH7. SRPM should be buildable on all Red Hat systems.

MD5 sums:
30d76eb29c75ca9bf5dcc4d4903de299  bash-3.0-29centos4_vulnfix.i386.rpm
89f0c72480a2dbe28d61503973e98443  bash-3.0-29centos4_vulnfix.src.rpm
57bb220cc9ac5ef2c445a4dece61814c  bash-3.0-29rh62_vulnfix.i386.rpm
4ab6aa1a5958da0e5290f43134b08f2a  bash-3.0-29rh62_vulnfix.src.rpm

If you want to be 100% sure that the code wasn't tampered with, build your own binary by using src RPM and verify that all patches apart from 140/141 are Red Hat original (140/141 were taken from Oracle's bash patches).

fix for Linux Skype 4.3 crash on startup

— Posted by hazard @ 2014-08-11 10:47
If you upgraded your Linux Skype to 4.3 and face a crash immediately after startup, the fix that worked for me is as follows:

  • Make of a backup of your home .Skype directory
  • Install sqlite package on your system if it isn't there already
  • Run: sqlite3 ~/.Skype/[YOURUSER}/main.db
  • DELETE FROM Messages WHERE type=68;
  • .quit
You will loose your file transfer history, but chat history will still be there. I found this workaround here. If it doesn't help, you may have to delete/rename your .Skype directory.

If you don't have in audio in Adobe Flash in Fedora 20, pulseaudio is the reason

— Posted by hazard @ 2014-06-27 19:15
If on your Linux PC YouTube wouldn't play more than one second of videos, while flash on other websites has no sound, or you have any other issues with audio in other apps, the likely culprit is Pulseaudio. I could never figure out the reason why the world needed pulseaudio, apart from the fact we have to use everything that Lennart Poettering creates.

Anyway, solution to the pain is easy, just follow excellent 30-second instruction on Mondo Grigio blog.

Sample jQuery.sheet online spreadsheet backend with load/save functionality

— Posted by hazard @ 2014-05-25 12:30
I've been integrating an online spreadsheet functionality into DokuWiki and jQuery.sheet looked like the most suitable candidate. It comes with a number of examples, but there is no server-side backend that will save/load the sheets. As a proof of concept I created a simple jQuery.sheet database backend example in Perl.

Fix for Fedora USB disks going into /run/media/[user]

— Posted by hazard @ 2014-02-03 00:27
As part of systemd/DBus revolution, newer Fedoras have this annoying feature that all USB disks get mounted into to /run/media/[user]/[diskname] (also /var/run/media, which is a symlink).

After some digging, I found that this is done supposedly for security reasons and being taken care of by udisks2 daemon. I'm still not sure what extra security this gives since 99% of Fedora desktops are not accessed by untrusted users, while admins of untrusted multiuser machines usually understand the risks. Nevermind. What I'm sure though, is that it breaks guest access to USB disks from Samba and breakes old scripts which were using /media.

To return old behavior back and make udisks2 change mount point to /media, create a file /etc/udev/rules.d99-usb-shared-media.rules:


udev should notice and read this file automatically. After this, just unplug and plug your USB drive back to see it in good old /media.

Smokeping Juniper JunOS plugin with routing-instance and logical-system support

— Posted by hazard @ 2013-12-01 19:07
I have hacked together a Smokeping plugin for Juniper JunOS devices that supports VRFs (routing-instance) and logical systems.
  • Download the plugin to your server.
  • Install by copying the file into lib/Smokeping/probes directory under your smokeping installation (e.g. to /opt/smokeping/lib/Smokeping/probes).
  • You might also have to install Net::OpenSSH Perl module, if it's not already installed (check by running "perl -e 'use Net::OpenSSH'").
  • Add the following section to your smokeping config:
    + OpenSSHJunOSPing
    packetsize = [in JunOS 1472 is the max for 1500 L3 MTU]
    junospass = [pass]
    junosuser = [user]
    # feel free to change params below as you wish
    forks = 5
    offset = 50%         
    timeout = 15
    step = 120
  • Individual targets are configured as follows:
    ++ sample-target
      probe = OpenSSHJunOSPing
      menu = [menu name]
      title = [title]
      host = [destination IP to ping from JunOS device]
      pings = [numer of ICMP pings to send, e.g. 5]
      source = [JunOS device to login into]
      logicalsystem = [logical system name, optional]
      vrf = [routing-instance name, optional]
  • ssh to the JunOS device once from the commmand line from the account of the user who is running smokeping (su -s /bin/sh [username]). On the first connect ssh will ask to add the new host to its known_hosts file, confirm it. Otherwise Smokeping will fail to login as the ssh key of your JunOS box is not in the known_hosts file.

OSPF: a protocol from hell, or Type 5 vs Type 7 LSAs

— Posted by hazard @ 2013-01-12 02:13
I've always thought that OSPF was one of the rotten tomatoes in the generally nicely-looking suite of core IP protocols. Its architecture is simply not suited well for real-life carrier networks. OSPF tries to impose its own rules on how the network should be built, instead of providing flexibility to adjust the protocol to the network. However, instead of throwing OSPF out and concentrating on a better IGP (e.g. something EIGRP-like), Internet/IETF community kept adding more and more band-aids to it. As a result, we've got a number of monstrous and complex specifications, which even vendors don't fully understand, resulting in bugs and incompatibilities between implementations, as well as lots of confusion to engineers.

A few days ago I got hit by another case which re-enforced my beliefs. I had to implement an OSPF network which had both Type 5 LSA E2 and Type 7 N2 routes for the same prefix. Moreover, there was a mix of IOS and JUNOS speakers in the same network. So, what is the route selection algorithm in that scenario? Google and you'll find at least three different answers. An incorrect answer from one of very reputable sources will say that E2 route will win over N2 no matter what the cost is. Another answer is that E2 route will be preferred if it has the same or lower metric than N2 route. And the third answer is the opposite: N2 route will win over E2 if it has the same or lower metric.

Well, to ease the pain of future generations, I'll say that the correct answer is that N2 route will win unless E2 has lower cost. That is, unless that future generation lives in a world where they have released an RFC to supersede RFC 3101, since things might change the same way as they did since RFC 1587, which apparently specified opposite behavior. If you're using Cisco IOS, it depends on which IOS you use, many IOSes will prefer E2, as they follow the behavior from RFC 1587, even though they were released much later than RFC 3101. Other IOSes will prefer N2.

I rest my case.

Long live OSPF, the reason for late-night maintenance headaches! As well as an additional source of revenue for network engineer certifications. :)

Forcing Fedora's preupgrade to use servers in Europe instead of Asia

— Posted by hazard @ 2012-01-07 19:23
I've decided to upgrade my FC14 to FC16. Along the way, I decided to do it using a method that I've never used before - preupgrade. Supposedly it's one of the easiest and less time consuming methods. Not in Cyprus ... (More)

nginx as protection against DDoS to Apache

— Posted by hazard @ 2011-08-28 18:33
A few days ago I was asked to help with a DDoS attack against a website. The DDoS itself was pretty generic, a small zombie network hammering particular URLs from the websites with GET requests. The websites were running on Apache, and despite that the target page was static, the DDoS was bringing Apache to its knees. System administrators tried to utilize various Apache modules and configuration tricks to protect against DDoS, but to no avail.

There was only one solution to this on my mind - install nginx. And that really helped. nginx is asynchronous by design and therefore handles load much much better. Whilst Apache was failing with several hundred simultaneous connections, nginx easily scaled to 10k caused by DDoS, whilst using only 20% CPU.

The first website was completely moved to nginx, with PHP being served through PHP/FastCGI. For the second website, the nginx was configured in proxy mode, so that it would forward all requests to the Apache, whilst enforcing limits against DDoS - 1 unique page request per IP per second, as well as blocking certain user agents. Below is an example configuration I created, relevant for CentOS/RHEL.

Fixing Greenplum 'unresolved in-doubt transaction' errors

— Posted by hazard @ 2010-06-29 05:00
We had an issue with a database server running Greenplum (commercial Postgresql for large-scale datawarehousing). Greenplum was starting, but attempts to do anything in the database were resulting in the following errors:

INFO: Crash recovery broadcast of the distributed transaction 'Commit Prepared' broadcast succeeded for gid = 1265880453-0032866370.
INFO: Crash recovery broadcast of the distributed transaction 'Abort Prepared' broadcast succeeded for gid = 1265880453-0032866371��C
psql: FATAL: DTM Log recovery failed. There are still unresolved in-doubt transactions on some of the segment databaes that were not able to be resolved for an unknown reason. (cdbtm.c:2829)
DETAIL: Here is a list of in-doubt transactions in the system: List of In-doubt transactions remaining across the segdbs: ("1265880453-0032866371��C" , )
HINT: Try restarting the Greenplum Database array. If the problem persists an Administrator will need to resolve these transactions manually.

Of course, manuals/forums/Google did not provide any useful ideas as regards to how 'to resolve these transactions manually'. Morever there was no backup handy (and the db was huge). I didn't care about two lost transactions, I just wanted to start the database. After an hour of attempts, eventually I succeeded. The trick was to delete files from pg_twophase/ subdirectories.

I'm blogging this in the hope that when somebody else faces this problem, he would be able to find this post through Google, saving his nerves. :-)

Guaba Beach Bar Limassol

— Posted by hazard @ 2010-06-23 14:33
Great music but a boo for customer service. I don't like being told "I don't give a shit" by supervisors. Having spent there a few hundred euros in a week, the least I want is to be handled politely. Fix yourself or you'll end up only with drunken kids. The atmosphere has been degrading ever since the move to a new place.

What a wonderful day ... NOT

— Posted by hazard @ 2010-03-19 14:39
It's been a while I posted something here, so as well it might be a rant.

  • Confortel Atrium hotel in Madrid tried to put 60 EUR worth of bar/restaurant charges on me, ignoring the fact that they were dated BEFORE the date I arrived.
  • Aegean airlines screwed up web check-in Madrid-Athens-Larnaca. I managed to check in for Madrid-Athens, but then the system would not let me check-in to Larnaca.
  • Aegean airlines did not manage to register me to Larnaca even in Madrid airport registration desk. "We use different systems". It should be mentioned that my transfer time in Athens was planned to be just one hour, which is already tight, without having to obtain a boarding pass...
  • ... and then the flight to Athens was delayed.
  • The duty-free in Madrid didn't put the bottle of wine I purchased in a sealed bag. So the bottle was happily confiscated by Athens airport security.
  • My passport got damaged (peeled off) just on the place where my photo is.
  • It seems I caught a cold.

Well, it could be worse, right ??

Kernel 2.6.28 for Fedora Core 8

— Posted by hazard @ 2009-02-28 14:31
In case someone wants to run a recent Linux kernel on an FC8 box, I have made an RPM for and you can download it here. Should also install on CentOS 5/RHEL 5 if you use --force.

Waltz with Bashir

— Posted by hazard @ 2009-01-24 17:34
Just finished watching Waltz with Bashir. Emotional, original and very beautiful. Not for the faint of heart.

I hope that it will get the well-deserved Oscar in the best foreign movie category.

You think that SPAM is distributed? I did so too.

— Posted by hazard @ 2008-11-14 14:31
Shutdown of a single ISP, McColo, has reduced world spam levels by 70%. Check the full story here. Amazing.

Karcher RoboCleaner in da house :)

— Posted by hazard @ 2008-08-17 18:44
At last I received my Karcher RC 3000 RoboCleaner. My carpet already looks more vibrant than ever :) The thing goes around the apartment collecting the dust, finds the base, dumps the dust into a bag in the base, and then starts again from the beginning... until you stop it. Perfect. As long as it can find the base ;)


MAD Video Music Awards 2008

— Posted by hazard @ 2008-07-13 06:34
I caught a glimpse of MAD Video Music Awards 2008 and I must say I was impressed. Greeks managed to stage a world-class show, with a vivid scene setup and excellent camera work.

It was the first time I saw a live production of such quality from Greece. Eurovision 2006 doesn't count as it was supported by European Broadcasting Union.


HP F4180 printer/scanner/copier

— Posted by hazard @ 2008-06-22 10:09
I recently purchased HP F4180 printer/scanner/copier for 50 EUR. It is amazing how inexpensive these things are nowadays.

As regards to Linux compatibility, I must say that it is very good and everything worked from the first attempt. Simply download HPLIP and all configuration is done automatically (at least on FC8). Printing works using CUPS, scanning using xsane. Thumbs up to HP!

New era in Cyprus telecoms

— Posted by hazard @ 2008-04-18 13:39
PrimeTel is building its own submarine fiber cable landing station and will bring one of the biggest international submarine cable systems to Cyprus. If you think that your connection is slow compared to what people in rest of the Europe get, wait until 2009 :)

Reliance Globalcom to land undersea NGN on PrimeTel station

Knockin' On The Heavens Door

— Posted by hazard @ 2008-04-06 14:05
If I'm asked about my favourite movie, my current answer would be "Knocking' On The Heaven's Door". Great script with excellent combination of drama, humour and action. Plus a soundtrack which perfectly fits the picture. 10/10.

Did you know ...

— Posted by hazard @ 2008-04-01 06:14


IBM-Lenovo X61

— Posted by hazard @ 2008-03-15 09:05
My colleague received an IBM-Lenovo X61 laptop, and we found that built-in Intel 4965 wireless is very slow when connected to 802.11b access point. Same problem persisted both with Fedora Core 8 and Ubuntu (iwlwifi driver).

Usual tweaking of ACPI and APIC parameters didn't help. Googling around showed a lot of people suffering from the same problem with Intel 4965 wireless cards. Eventually I resolved the problem by removing iwlwifi driver files from /lib/modules, and installing ndiswrapper (allows to use Windows network drivers) + Intel 4965 driver for Windows XP.

The Vista Experience

— Posted by hazard @ 2008-03-10 07:02
My old desktop machine had died, so I decided to buy a new laptop to replace it. Even though most of the time it will be sitting in the same place, laptops nowadays are cheap and mobility is a nice option to have.

So, I got an HP Compaq 6710b. Along the way I purchased upgrade to 2 GB RAM and 320 GB hard drive. The laptop by default comes with Vista, which I wanted to keep, just in case I need to run some Windows stuff which wouldn't work in Linux. At first I start the laptop with its factory 160 GB HDD: HP's Vista installer loads from a special partition and in less than two hours I have a running Vista (enough to install Fedora two times). Then I proceed to make recovery DVD, so that I can install Vista on the 320 GB disk. Vista needs only two DVDs and another hour to do that; excellent. I replaced the hard drive and started the whole Vista installation process again from the DVD.

Of course, HP's Vista installer takes over entire disk space without asking, I'm sure only to make the experience more user-friendly for the user (after all, these "Advanced" buttons are way too tricky). The thing is, I want to give only 50 GB to Vista. Anyway, Vista gets installed, and I'm logged in. I start the Disk Management tool - actually, I already got irritated at this point because Microsoft thought it is a good idea to break old ways of using Windows - and whoala, it tells me that I can shrink my disk to 160 GB. To those of you who don't know, actual space used by Vista files is less than 20 GB.

The tool also vaguely mentions that I can get rid of shadow copies and paging file in order to increase available space. Of course it is useless to use built-in Help to find how do I actually do that, but thanks to Microsoft's competitor Google, this information was retrieved and necessary actions were performed. Now, I can shrink my drive by 3-4 GBs more. Fantastic! Just what I dreamed of, to buy 320 GB HDD and leave 155 GB for Vista. Googling around shows that what I got is normal for Windows disk shrinking tool - it frees about 50% of space.

"Screw it!", I think to myself, and proceed to install Fedora 8. As a precaution, I left first partition empty at 60 GB, so that I can try to install Windows again later (of course, I was also understanding that doing standalone Vista/XP install would be painful because it would not have the HP drivers). Anyway, in one hour FC8 is up and running - "nohz=off" was needed to make the Fedora installer work.

However, built-in modem is NOT up and running, as it is one of the softmodems for which Agere has not released drivers. I find a few suggestions on the mailing lists that making a Frankenstein driver by copying .o files from one driver to another might work, but I only get a few OOPS'es as a result and NOT a working modem.

Now, I need the modem, because I have to connect to remote console servers over the phone for troubleshooting of network outages. So, I insert Vista rescue DVD again, hoping that I might have missed an option to install into a specific partition. Nope, not there. And it also overwrote my MBR without asking. How nice.

I look back at my 160 GB drive. What if I shrink that one and then transplant Vista to my 320 GB HDD? In a few minutes my 160 GB is back in the laptop, I run Windows shrinking tool, and whoala, it shrinked it down to a whopping 45 GB. A few minutes more, and 160 GB is in USB enclosure, 320 GB is back inside the laptop, and dd is happily copying first partition from 160 GB drive to 320 GB one. One hour passes; dd has finished, not as quickly as I expected - only 7 MB/s. Anyway, let's try to boot Vista... drums roll (in my head)... Vista's loading bar starts to run around the screen... KABOOM, "winload.exe is missing or corrupted".

Back to Linux, mount the Vista partition. winload.exe is there and MD5 is the same as the original one on the 160 GB HDD. Also I can't find boot.ini anywhere. Hmmm. Time for another visit to Microsoft's competitor Google. Aha - Microsoft is improving at friendly error messages: "winload.exe is missing or corrupted" actually means that disk ID has changed, and just to make it easier for the average Joe to use Vista, Microsoft has started checking that disk ID entered into the bootloader config matches the one on the actual drive. Otherwise it won't boot, even if everything else is in place.

Alright, let's see how we can fix that bootloader config. Another innovation! Finally Microsoft has managed to get rid of that prehistoric way of configuring bootloader using a text boot.ini file! Now we have a shiny new registry-like binary database somewhere else. To edit it, use BCDEDIT.EXE. Cool. My problem is that I don't have a working Vista to run it.

"If the mountain will not come to Mohammed, Mohammed must go to the mountain". Disk ID is written in the MBR. Armed with dd and mcedit in hex mode, I copy the Disk ID from the 160 GB HDD's MBR into 320 GB, then use fdisk to confirm that they match. Reboot, select "Other" in GRUB....


Looking back, I'm glad at how painless and inspiring my Vista experience was. Such experiences bring more users to Linux.

UPDATE: I later discovered existence of "ntfsresize" tool under Linux, which apparently does much better job at NTFS resizing than Vista's built-in one.