Perhaps I misunderstood you, but this is a critical point. A difference in price can be significant (statistically speaking) even if the change is only 0.1%, assuming your price distribution across the market is very tight, yet no one would notice such a change. Alpha levels have nothing whatsoever to do with determining whether the magnitude of a change will have detrimental effects.

We can't set an arbitrary number in the currently discussed case, but there are plenty of other metrics to look at if we were so inclined. We could model how much an X% increase in gas prices will affect GDP growth or unemployment or whatever, but at the end of the day we'd still need to pick a number. A better tool would be to look at the trends in the pricing information. If the short term volatility of the market is significantly more (as determined by real statistics) than the long-run volatility, we could characterize the change as 'significant', though we'd probably want to control for a number of factors (seasonal variations being a big one). If gas prices were jumping this much in the early 90s, it would almost certainly be significant, since volatility was extremely low. Yet nowadays it's not so clear, and a genuine mathematical analysis would be needed.

Anything short of this is pure speculation.