Phased Ubuntu to Debian migration of log collectors across three offices
How I moved a financial asset manager's log collectors to new Debian machines, one kind of log source at a time.
In plain terms
A log collector is a server that gathers logs from a client's systems and passes them on to us, so we can watch for security issues. One client, a financial asset manager, had collectors and forwarders in three offices, all running Ubuntu Linux, and they had to move to Debian, another version of Linux.
Not every source works the same way. Some logs are pulled over APIs, where we go and fetch them. Others are sent by the client's systems to a collector's IP address: its fixed address on the network. If the machine at that address stops answering, those logs have nowhere to go.
That's what made this migration delicate. Swapping machines had to happen without breaking the sources that point at a fixed IP, and if anything went wrong, there had to be a way back.
The problem
A financial asset manager's log collectors and forwarders had to move from Ubuntu to Debian across three offices. Some sources were pulled over APIs; others sent their logs to the collectors' fixed IPs.
What I did
I migrated in phases. First I moved the API-based sources to the new Debian machines.
For the sources that send to a fixed IP, the new machines took over the old ones' addresses using IP swap scripts that undo themselves unless the change is confirmed. The scripts have their own case study, next.
The outcome
Finished with minimal disruption, and none of the fixed-IP sources had to be reconfigured.