You won't be the only "Ultron does not exist" member of fandom. I fully support your decision.

Here's one of my favorite things about those spoilers: the family could be Barney's or a part of the visions Wanda gave the lot of them. That's where most of the fics went pre-movie.