Why does the significance level change between being significant and non-significant?

When a test goes live and there is low traffic to the site, each session will have a greater impact on the significance level than it will have in the long run. You should therefore let the test be active for at least 1-2 weeks before taking the significance level a bit more seriously.