I think some of the spoilers sort of already confirmed (eh...) Cas was yanked out? That's the idea I have at this point. However, at the time, I didn't interpret there was any sure indication in those scenes that Cas was out of there.

Also, the tone I've seen used by other fans in other places is that only a delusional casfan would think Cas and Dean were both there all by their lonesome. Since you're not a Casfan yourself, I guess it's confirmation this is not the case, though: a large portion of fandom thought they were both were stuck there, regardless of affiliation.