The infants died at two Northern Ireland hospitals in December 2011 and January 2012.
A statement from the PSNI said the families have been notified of the development.
It read: "Police are conducting preliminary investigations into the circumstances surrounding the deaths of four babies from pseudomonas at two Northern Ireland hospitals in December 2011 and January 2012. Two families have made complaints to police but the families of all four babies have been made aware of the situation."
The first death of the pseudomonas outbreak occurred at Altnagelvin Hospital in Co Londonderry in December 2011.
Three more babies died from a different strain of pseudomonas at Belfast's Royal Jubilee Maternity Unit the following month.
An investigation by the Regulation and Quality Improvement Authority (RQIA) traced the deadly bug to contaminated taps in neo-natal units.
The RQIA was also critical of a lack of communications between health official and said some of the deaths could have been prevented if the Belfast Health Trust had acted sooner.