A recent interaction with AWS support reminded me of something that happened to me and a co-worker some years ago in a previous job. The people and companies in the story have been anonymized, but I’m “Eric”.
I (“Eric”) worked for the online development team of a large media company (The Company) “somewhere in Europe”. Some years ago our IT infrastructure was outsourced to India to The Service Provider, since “it’s cheaper to have them maintain our infrastructure”.
For a website migration one of our consulting firms needed a new web server. Previously we would have set up the server ourselves, but now we were supposed to request that The Service Provider set it up for us. While we were somewhat skeptical, we decided to try and play by the new rules.
The story starts after Isaac (person #1) from The Consulting Firm let us know that they need a web server for the migrated site.
April 16th (day 1):
- Trevor (from The Company, person #2) notifies Isaac that the servers can now be ordered and that we need the server specifications.
- Isaac delivers server configuration details to Trevor
- Trevor delivers server specifications to Thomas (from The Service Provider, person #3)
- Thomas asks if there’s a deadline for the installation
- Trevor replies that the only deadline is the shutdown of the old servers on June 15th
- Thomas delivers server specifications to Ayush (The Service Provider, person #4). No GUI is requested.
- Ayush replies that servers will be ready May 1st
- Trevor notifies The Consulting Firm that the servers will be delivered by May 1st, so installations could get started by May 6th.
- Ayush asks if The Company wants a GUI.
- Trevor replies no.
April 24th (day 9):
- Vijay (The Service Provider, person #5) notifies Trevor that the server file systems were configured in a way different from Trevor’s request
- Trevor replies with a request to configure file system as originally requested
- Vijay replies with a request for a teleconference
- Ayush tells Vijay to follow Trevor’s initial request for filesystem configuration
- Ayush asks if ports other than 22 are needed
- Thomas asks if servers are ready to be delivered
- Ayush notes that if port 22 needs to be open, they’ll need approval from Jack (The Company, person #6)
May 5th (day 20):
- Ayush delivers a list of server details to Trevor
- Thomas asks Trevor if there has been any progress
- Thomas asks Ayush to call up a meeting to discuss operations with Trevor
- Trevor asks for login credentials for the servers and for some network configuration changes to allow admin access to the servers from The Company’s and The Consulting Firm’s networks
- Vijay sends the root credentials to Trevor in a plain-text email
- Mehul (The Service Provider, person #7) asks for the network configuration change request in an Excel template
- Trevor tells The Consulting Firm that the servers won’t be ready on the 11th, but maybe on the 13th. This turns out to be overly optimistic.
- Cameron (Hosting company for the old servers, person #8) reminds Thomas that there will be a datacenter move on June 15th
- Trevor tells Mehul that he can’t properly fill in the Excel because all the required information doesn’t exist yet. Trevor repeats the required network changes in the email.
- Thomas asks Mehul and Ayush if things are progressing
- Mehul says they’re waiting for source IP (which Trevor sent on the 8th)
- Thomas asks who he’s waiting for
- Mehul says Trevor, and claims that Trevor has requested access from the public Internet, not a specific IP range
- Mehul asks Jack for help
- Thomas tells Ayush and Mehul that if their work is pending on something, they should work towards getting that obstacle solved
- Trevor replies with a copy of previous instructions where he requested opening access from The Company’s and The Consulting Firm’s networks
May 15th (day 30):
- Mehul says they’ve allowed access as requested
- Trevor says he can’t access from The Company network or the The Consulting Firm’s network, and requests opening access from The Company’s network again
- Mehul tells Trevor that The Consulting Firm should try from the Internet, and claims that The Company’s connection is blocked in The Company’s firewall
- Trevor re-iterates that The Consulting Firm can’t connect via Internet and that The Company’s network does not block any outgoing traffic
- Jack gives Mehul instructions on how to investigate the network problem
- Mehul asks Trevor to try connections again
- Trevor replies that some connections work from private IPs, but not all
- Mehul replies with a list of server IPs and instructions to use private IPs
- Trevor re-iterates that some connections work from private IPs, but not all. Also that The Consulting Firm is unable to connect at all.
- Ayush asks if the connections are fixed now
- Trevor reminds Thomas that time is running out
- Thomas replies that changing firewall settings for The Company got stepped over by other things on The Service Provider’s side and that he’s escalated the issue
- Mehul asks Trevor to try the public IP again and claims that the private IP address is blocked by The Company
May 25th (day 40):
- Trevor replies that it still does not work
- Mehul asks Trevor to try again, having now personally tested the connection from Internet.
- Trevor replies that he’s waiting for word from The Consulting Firm
- Trevor confirms that the connection does indeed work from Internet, and that The Consulting Firm can now proceed with their work.
- Mehul asks if Trevor needs access through The Company network.
- Trevor replies that the connection from The Company network works fine now.
- Ayush requests a meeting with Trevor about support handover to operations
- Trevor asks what support is this referring to
- James (The Company, person #9) replies that it’s about general infrastructure support
June 5th (day 51):
- Trevor notifies Ayush and Mehul that server network configurations were incorrect. Also that after fixing the configuration and rebooting the server The Consulting Firm can no longer login to the server because the password no longer works.
- Ayush instructs Vijay to “setup the repository ASAP” [?]
- Vijay responds that “licences are not updated for The Company servers” [??]
- Vijay sends original root credentials in a plain text email again [???]
- Thomas reminds Ayush and Mehul that the servers need to be moved by the 12th
- Trevor reminds Thomas that the deadline was extended by Trevor to the end of June the previous week
- Trevor replies to Vijay that the original credentials sent no longer work
- Vijay asks Trevor to try again
- Mehul asks for the details of the unreachable servers, which were mentioned in the previous email
- Trevor sends a summary of current status (can’t access from The Company’s network again, server passwords not working) to Thomas, Ayush, Mehul and others.
- Vijay replies “Can we discuss on this”
- Trevor replies that he’s always reachable by Skype or email
- Mehul says that access to private IPs is not under his control. “Looping John and Jared”, but no such people added to recipient list. Mehul repeats that from The Company’s network private IPs should be used.
- Thomas tells Trevor that the issue has been escalated again on The Service Provider’s side
- Thomas complains to Roger (The Service Provider, person #10), Theodore (The Service Provider, person #11) and Matthew (The Service Provider, person #12) that the process isn’t working
- Theodore asks Peter (The Service Provider, person #13), Mehul and Vinod (The Service Provider, person #14) what is going on
- Peter responds that websites should be implemented using Netscaler, and asks no-one in particular if they could fill an Excel template
- Theodore asks who should be filling out the template
- Trevor asks Thomas if he still thinks the sites can be in production by the latest deadline, June 29th, and if he should install the server on AWS instead
- Thomas asks Theodore if configuring the network really takes two weeks, and basically tells the team to try harder
- Theodore replies that configuring the network doesn’t take two weeks, but getting the required information for that often does. Also that there are resourcing issues related to such configurations.
- Thomas suggests a meeting to fill the template
- Thomas asks if there’s any progress
- Ayush replies that if The Company provides the web service name, The Service Provider can fill out the rest
- Trevor delivers a list of site domains and required ports
- Thomas forwards the list to Peter
- Tyler informs Trevor that any AWS servers should be installed by Another Service Provider
- Trevor explains that the idea was that he would install the server on The Company’s own AWS account
- Paul (The Company, person #15) informs Trevor that all AWS server installations are to be done by Another Service Provider, and that they’ll have time to do it in August
- Kane (The Company, person #16) asks for a faster solution, as they’ve been waiting for nearly two months already
- Peter replies that he needs a list of fully qualified domain names, instead of just the site names
- Trevor delivers a list of current blockers to Thomas, Theodore, Ayush and Jagan (The Service Provider, person #17)
- Ayush instructs Vijay and the security team to check network configuration
- Thomas reminds Theodore, Ayush and Jagan to solve the issues, and reminds them that the original deadline for this was May 1st
- Theodore informs everyone that the servers’ network configuration wasn’t compatible with the firewall’s network configuration, and that Vijay and Ayush are working on it
June 15th (day 61)
- Peter asks Thomas and Ayush if they can get the configuration completed tomorrow
- Thomas asks Theodore, Ayush and Jagan if the issues are solved
- Ayush tells Trevor that they’ve made configuration changes, and asks if he can now connect
- Trevor replies to Ayush that he still has trouble connecting to some of the servers from The Company’s network
- Trevor delivers network configuration details to Peter
- Ayush tells Vijay and Jai (The Service Provider, person #18) to reset passwords on servers so Trevor can log in, and asks for support from Theodore with network configurations
- Matthew replies that Theodore is on his way to The Company
- Vijay resets the password and sends it to Ayush and Jai
- Ayush sends the password to Trevor via plain-text email
- Theodore asks Trevor and Ayush if the problems are resolved
- Ayush replies that connection from The Company’s network does not work, but that root password was emailed.
June 18th (day 64)
- Tyler (The Company, person #19) sends email to everyone and cancels the migration
MEANWHILE, on June 11th:
- Eric and Trevor set up the server on The Company’s AWS account before lunch, and deliver it to the Consulting Firm. 😉