I’m not familiar with the building, but it looks like it could be limestone, which is porous. That would make it difficult to remove something painted on without removing a decent amount of material from the surface. The paint is probably in there pretty good, and power washing may have pushed it deeper.