This morning, I was awakened at 2AM by a call from Nino. He was on standby providing support to the deployment team and it seems there’s a problem. I quickly dialed-in to the conference call and got the log report. I saw, as Nino already did, that there’s a communication problem between two components, the CWS and the PIS.
However, due to the configuration of the production environment, it is hard to confirm. There is a staging environment but the configuration is different (bad) so it’s pretty much useless. Eventually, the deployment team did manage to isolate one production server and we tested on it.
Same conclusion: there is a communication problem between CWS and PIS. I asked if PIS is available on the port the CWS is trying to connect to. That probably switched the light bulbs in the deployment team’s collective heads because they instead of responding, they configured something, and voila, it worked!
UPDATE: I found out that the port our system has been configured to use is not compatible in production because they had decided to keep the old application server up (good) on that port and have the new application server on a different port (but they hadn’t informed us, bad).